eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working on a project in VSCode that has multiple directories, each of which requires a different Python virtual environment. My virtual environments are located in the ~/.virtualenvs directory and my workspace is structured like this: ~/.virtualenvs/ │ ├── venv_A/ │ └── venv_B/ my_workspace/ │ ├── project_A/ │ └── script_A.py │ └── project_B/ └── script_B.py  I want VSCode to automatically switch to the appropriate virtual environment (venv_A for project_A, and venv_B for project_B) located in ~/.virtualenvs when I open a Python file from each directory within the workspace. Currently, I have to manually select the virtual environment through the command palette each time. I have tried looking through the VSCode documentation and searched for guides or tutorials on how to achieve this functionality, but I haven't found anything that addresses this specific issue. I expected there to be some configuration options either through the .vscode/settings.json file or the workspace settings that would allow me to specify which virtual environment should be used for each directory, and how Pylance, pylint, and yapf should adapt accordingly. I am aware that VSCode has support for workspaces and .env files, but I'm not sure how to configure it to auto-switch virtual environments based on the directory, and to have Pylance, pylint, and yapf adapt accordingly. I also found the issue Select pyenv environment based on folder .python-version file that is not closed.

Answer: The easiest way is to open project_A and project_B as workspaces respectively, and then select an interpreter for the workspace, vscode will remember your choice, and will still use the previously selected interpreter when it is opened next time. Another approach is to use Multi-root Workspaces  Open a new window and use Add Folder to Worspace... to add both folders to the current workspace,    Then created separate virtual environments for both folders using Ctrl+Shift+P --> Python: Create Environment... and select the interpreter for the respective folder.  You can also select the .py file in the corresponding folder, and then click the python version in the lower right corner to switch   These documents may also be useful:  https://code.visualstudio.com/docs/python/python-tutorial#_select-a-python-interpreter  https://code.visualstudio.com/docs/python/environments#_working-with-python-interpreters  https://code.visualstudio.com/docs/editor/profiles
Question: Let's say I have the following list: l = [ {"a": 10, "b": 100, "c": 100}, {"a": 20, "b": 100, "c": 100}, {"a": 30, "b": 100, "c": 100}, ]  I know "a" is unique in each item: assert len({x["a"] for x in l}) == len(l)  I want to generate a mapping of the "a" value to the rest of each item so my end result is the following dictionary: { 10: {"b": 100, "c": 100}, 20: {"b": 100, "c": 100}, 30: {"b": 100, "c": 100}, }  So far I've come up with the following: {x["a"]: {k: v for k, v in x.items() if k != "a"} for x in l}  Is this the best way to write this? Or is there a better way or a built in function that I'm missing?

Answer: Maybe dict.pop is what you want? out = {d.pop('a'): d for d in l} print(out)  Prints: {10: {'b': 100, 'c': 100}, 20: {'b': 100, 'c': 100}, 30: {'b': 100, 'c': 100}}
Question: When applying the Python function type() to a single object, the type of the object is returned. num = 5 type(num) Out[1]: int  When embedding this output into a string and printing the result, this seems to behave as expected. num = 5 print(f"type of {num} is {type(num)}") type of 5 is <class 'int'>  However, when using this exact message as an assertion error message, the type disappears from the message output. num = 5 assert isinstance(num,str), f"type of {num} is {type(num)}" AssertionError: type of 5 is  I am running the code through a notebook on a Databricks cluster, which is displayed in a web browser. The expected output would be: AssertionError: type of 5 is <class 'int'>. What is the reason for this and how can it be avoided? Edit: As mentioned by several commenters, the code works just fine running outside of the databricks environment. I have successfully verified this too using Python 3.10.6.

Answer: The character < in the output is being interpreted as the beginning of an HTML tag. By applying replace("<","&lt;") to the f-string, this can be avoided. The full code then becomes: num = 5 assert isinstance(num,str), f"type of {num} is {type(num)}".replace("<", "&lt;")
Question: I'm having problems with a dataset I want to use to display on a PowerBI report but in some registers it has a start time and an end time with a different date, which makes me hard to display on a daily basis. I want to divide the register automatically for each day. I have the following register, for example:     Date_Start Date_End     18/04/2023 10:53:00 a. m. 20/04/2023 03:51:00 a. m.     Since I am using Date_Start column to create the report on a daily basis, with this register, I can't display the date 19/04/2022, since I don't have a register on Date_Start with that date. So, I want to process the register and divide it like this:     Date_Start Date_End     18/04/2023 10:53:00 a. m. 18/04/2023 11:59:59 p. m.   19/04/2023 00:00:00 a. m. 19/04/2023 11:59:59 p. m.   20/04/2023 00:00:00 a. m. 20/04/2023 03:51:00 a. m.     I am not sure if this is possible using PowerQuery or maybe using Python among Pandas or Numpy library. Can you support me with this topic?, I'd appreciate it. Thanks! :)

Answer: Powerquery method let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content], #"Added Custom" = Table.AddColumn(Source, "Custom", each List.Transform({Number.IntegerDivide(Number.From([Date_Start]), 1)..Number.IntegerDivide(Number.From([Date_End]), 1)}, each Text.From(Date.From(_)))), #"Expanded Custom" = Table.ExpandListColumn(#"Added Custom", "Custom"), #"Added Index" = Table.AddIndexColumn(#"Expanded Custom", "Index", 0, 1, Int64.Type), #"Added Custom1" = Table.AddColumn(#"Added Index", "Date_Start.", each if [Index]=0 then [Custom] & " " & Text.From(DateTime.Time([Date_Start])) else [Custom] & " " &"00:00 AM"), #"Added Custom2" = Table.AddColumn(#"Added Custom1", "Date_End.", each if [Index]=List.Max(#"Added Index"[Index]) then [Custom] & " " & Text.From(DateTime.Time([Date_End])) else [Custom] & " " &"11:59 PM"), #"Removed Columns" = Table.RemoveColumns(#"Added Custom2",{"Date_Start", "Date_End", "Custom", "Index"}), #"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Date_Start.", type datetime}, {"Date_End.", type datetime}}) in #"Changed Type"
Question: I read that vars() is a built-in that returns the __dict__ attribute of class, module or object. But when I checked for vars(Person) is Person.__dict__, it returned False(Person is the name of class). class Person: def __init__(self, name, age): self.name = name self.age = age vars(Person) is Person.__dict__ # False 

Answer: Classes create a new mappingproxy on every __dict__ access. You would see the exact same results from Person.__dict__ is Person.__dict__.
Question: I am using pandas to read csv on my machine then I create a pyspark dataframe from pandas dataframe. df = spark.createDataFrame(pandas_df)  I updated my pandas from version 1.3.0 to 2.0 Now, I am getting this error:    AttributeError: 'DataFrame' object has no attribute 'iteritems' 

Answer: Found an answer on github: https://github.com/YosefLab/Compass/issues/92 It is an issue going on. iteritems is removed from pandas 2.0 For now I need to downgrade pandas back to version 1.5.3  Edit: Other workarounds may be Use the latest Spark (3.4.1) https://spark.apache.org/downloads.html  For pandas >=2.0 You can also assign DataFrame.items to DataFrame.iteritems import pandas as pd pd.DataFrame.iteritems = pd.DataFrame.items  https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.items.html?highlight=items#pandas.DataFrame.items
Question: I am trying to create a virtual environment for python 2.7 on Ubuntu 22.04. I always receive an error as follows: RuntimeError: failed to query /usr/bin/python2.7 with code 1 err: ' File "/usr/local/lib/python3.10/dist-packages/virtualenv/discovery/py_info.py", line 152\n os.path.join(base_dir, exe) for exe in (f"python{major}", f"python{major}.{minor}")\n ^\nSyntaxError: invalid syntax\n' Here is a capture of my terminal for useful information: user@machine:~/environments$ ls /usr/bin/pytho* /usr/bin/python2 /usr/bin/python2.7 /usr/bin/python3 /usr/bin/python3-config /usr/bin/python3.10 /usr/bin/python3.10-config user@machine:~/environments$ pip --version pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10) user@machine:~/environments$ virtualenv --version virtualenv 20.23.0 from /usr/local/lib/python3.10/dist-packages/virtualenv/__init__.py user@machine:~/environments$ virtualenv -p /usr/bin/python2.7 py2_env RuntimeError: failed to query /usr/bin/python2.7 with code 1 err: ' File "/usr/local/lib/python3.10/dist-packages/virtualenv/discovery/py_info.py", line 152\n os.path.join(base_dir, exe) for exe in (f"python{major}", f"python{major}.{minor}")\n ^\nSyntaxError: invalid syntax\n' user@machine:~/environments$  Has anyone else had this problem, or successfully achieved this?

Answer: virtualenv versions >= 20.22.0 dropped support for creating Python environments for Python versions <= 3.6, so you'll need to downgrade virtualenv, e.g.: pip install virtualenv==20.21.1
Question: Whats the recommended way to define an output schema for a nested json, the method I use doesn't feel ideal. # adding to planner -> from langchain.experimental.plan_and_execute import load_chat_planner refinement_response_schemas = [ ResponseSchema(name="plan", description="""{'1': {'step': '','tools': [],'data_sources': [],'sub_steps_needed': bool}, '2': {'step': '','tools': [<empty list>],'data_sources': [<>], 'sub_steps_needed': bool},}"""),] #define json schema in description, works but doesn't feel proper refinement_output_parser = StructuredOutputParser.from_response_schemas(refinement_response_schemas) refinement_format_instructions = refinement_output_parser.get_format_instructions() refinement_output_parser.parse(output)  gives: {'plan': {'1': {'step': 'Identify the top 5 strikers in La Liga', 'tools': [], 'data_sources': ['sports websites', 'official league statistics'], 'sub_steps_needed': False}, '2': {'step': 'Identify the top 5 strikers in the Premier League', 'tools': [], 'data_sources': ['sports websites', 'official league statistics'], 'sub_steps_needed': False}, ... '6': {'step': 'Given the above steps taken, please respond to the users original question', 'tools': [], 'data_sources': [], 'sub_steps_needed': False}}}  it works but I want to know if theres a better way to go about this.

Answer: From what I can see the recommended approach is to use the pydantic output parser as opposed to the structured output parser... python.langchain.com/docs/modules/model_io/output_parsers/… (and dealing with nesting explained here... youtube.com/watch?v=yD_oDTeObJY). e.g. from langchain.output_parsers import PydanticOutputParser from pydantic import BaseModel, Field, validator from typing import List, Optional ... class PlanItem(BaseModel): step: str tools: Optional[str] = [] data_sources: Optional[str] = [] sub_steps_needed: str class Plan(BaseModel): plan: List[PlanItem] parser = PydanticOutputParser(pydantic_object=Plan) parser.get_format_instructions()
Question: I can see everything but the Embedding of the documents when I used Chroma with Langchain and OpenAI embeddings. It always show me None for that Here is the code: for db_collection_name in tqdm(["class1-sub2-chap3", "class2-sub3-chap4"]): documents = [] doc_ids = [] for doc_index in range(3): cl, sub, chap = db_collection_name.split("-") content = f"This is {db_collection_name}-doc{doc_index}" doc = Document(page_content=content, metadata={"chunk_num": doc_index, "chapter":chap, "class":cl, "subject":sub}) documents.append(doc) doc_ids.append(str(doc_index)) # # Initialize a Chroma instance with the original document db = Chroma.from_documents( collection_name=db_collection_name, documents=documents, ids=doc_ids, embedding=embeddings, persist_directory="./data") db.persist()  when I do db.get(), I see everything as expected except embedding is None. {'ids': ['0', '1', '2'], 'embeddings': None, 'documents': ['This is class1-sub2-chap3-doc0', 'This is class1-sub2-chap3-doc1', 'This is class1-sub2-chap3-doc2'], 'metadatas': [{'chunk_num': 0, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'}, {'chunk_num': 1, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'}, {'chunk_num': 2, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'}]}  My embeddings is also working fine as it returns: len(embeddings.embed_documents(["EMBED THIS"])[0]) >> 1536  also, in my ./data directory I have Embedding file as chroma-embeddings.parquet  I tried the example with example given in document but it shows None too # Import Document class from langchain.docstore.document import Document # Initial document content and id initial_content = "This is an initial document content" document_id = "doc1" # Create an instance of Document with initial content and metadata original_doc = Document(page_content=initial_content, metadata={"page": "0"}) # Initialize a Chroma instance with the original document new_db = Chroma.from_documents( collection_name="test_collection", documents=[original_doc], embedding=OpenAIEmbeddings(), # using the same embeddings as before ids=[document_id], )  Here also new_db.get() gives me None

Answer: You just need to specify that you want the embeddings as well when using .get # Get all embeddings db._collection.get(include=['embeddings']) # Get embeddings by document_id db._collection.get(ids=['doc0', ..., 'docN'], include=['embeddings'])
Question: In Kivy, the widgets' color property allows enter its value as a string of a color name, too, e.g. in .kv file:  Label: color: "red"  Is there a list of all possible color names?

Answer: TL;DR from kivy.utils import colormap # import dict with all CSS 3 Colors # like {'aliceblue':[0.9411764705882353, 0.9725490196078431, 1.0, 1.0]} 'aliceblue' in colormap # True  For the plot, see the end of this answer.  Dictionary of colors The kivy docs mention that colors referenced by name are retrieved from an object called colormap. This object resides in kivy.utils as a variable storing a dictionary comprehension that iterates over another dictionary named hex_colormap: colormap = {k: get_color_from_hex(v) for k, v in hex_colormap.items()}  The ultimate source for these dictionaries is referenced only indirectly in the docs (link). A better reference would be: CSS 3 Colors (recommended by the W3C). At any rate, in order to retrieve all the valid color names, you can import either one of these objects: from kivy.utils import hex_colormap, colormap hex_colormap # name (key): hex (value) {'aliceblue': '#f0f8ff', 'antiquewhite': '#faebd7', ... 'yellow': '#ffff00', 'yellowgreen': '#9acd32'} print('aliceblue' in hex_colormap) # True (all colors in CSS3) colormap # name (key): rgba (value) {'aliceblue': [0.9411764705882353, 0.9725490196078431, 1.0, 1.0], 'antiquewhite': [0.9803921568627451, 0.9215686274509803, 0.8431372549019608, 1.0], ... 'yellow': [1.0, 1.0, 0.0, 1.0], 'yellowgreen': [0.6039215686274509, 0.803921568627451, 0.19607843137254902, 1.0]} print('rebeccapurple' in hex_colormap) # False (only color added in CSS4)  * on "rebeccapurple": Changes from Colors 3. Plot The matplotlib docs contain a nice function (plot_colortable) that you can copy/paste to plot a list of named colors. You can pass either one of the dictionaries to this function to get a nice sorted list of colors and their names. plot_colortable(colormap) # add `sort_colors=False` for unsorted plot plt.show()  Result:  Of course, this plot (showing Kivy's 147 named colors) is just the same as the plot already shown in the docs for mcolors.CSS4_COLORS (containing CSS4's 148 colors), the only difference being that my plot is missing "rebeccapurple".
Question: I'm trying to stream results from Open AI using a Lambda function on AWS using the OpenAI Python library. For the invoke mode I have: RESPONSE_STREAM. And, using the example provided for streaming, I can see the streamed results in the Function Logs (abbreviated below): Response null Function Logs START RequestId: 3e0148c3-1269-4e38-bd08-e29de5751f18 Version: $LATEST { "choices": [ { "finish_reason": null, "index": 0, "logprobs": null, "text": "\n" } ], "created": 1685755648, "id": "cmpl-7NALANaR7eLwIMrXTYJVxBpk6tiZb", "model": "text-davinci-003", "object": "text_completion" } { "choices": [ { "finish_reason": null, "index": 0, "logprobs": null, "text": "\n" } ],.... but, the Response is null. I've tested this by entering the URL in the browser and by performing a get request via cURL: both respond with null. Below is the exact code (with the secret key changed) that I used, but it can also be found on the link provided: import json import openai import boto3 def lambda_handler(event, context): model_to_use = "text-davinci-003" input_prompt="Write a sentence in 4 words." openai.api_key = 'some-secret key' response = openai.Completion.create( model=model_to_use, prompt=input_prompt, temperature=0, max_tokens=100, top_p=1, frequency_penalty=0.0, presence_penalty=0.0, stream=True ) for chunk in response: print(chunk) 

Answer: You are having trouble because python runtimes do not currently support streaming responses. From 4/7/2023 AWS announcement of streaming responses:  Response streaming currently supports the Node.js 14.x and subsequent managed runtimes.  As of 6/8/2023 this is still true.
Question: I am working with LangChain for the first time. Due to data security, I want to be sure about the storage of langchain's vector store storage. I am using HNSWLib vector store, which mentions it is an in-memory store. What does it mean? Does Langchain/vector stores store any data in its servers? https://js.langchain.com/docs/modules/indexes/vector_stores/integrations/hnswlib https://github.com/nmslib/hnswlib

Answer: HNSWLib store data in the server where the project is host. So if you host your server into vercel then your vector store is running in memory in vercel server. You can test this logic if true when you execute await vectorStore.save(directory); which will generate vector files inside your project directory.
Question: We are dealing with a very large legacy SQLAlchemy model (and underlying table schema) aggregating many logically-separate models, that cannot, for practical reasons, be refactored. We would want to be able to design authorizations/permissions on sub-models that restrict read/write to a subset of attributes (and perhaps methods as well). What would be the best approach to design a "partial view" Mixin/Class/Metaclass that would allow:  keeping the same original SQLAlchemy/flask-sqlalchemy model underneath (support .query etc) once loaded/inited, restrict access to a designated subset (stored at class level) of attributes  In essence going from: class MyLargeModel(db.Model): id = db.Column(db.Integer, primary_key=True) foo = db.Column(db.Text, nullable=False) bar = db.Column(db.Text, nullable=False) baz = db.Column(db.Text, nullable=False)  To something like: class ModelBase(db.Model): id = db.Column(db.Integer, primary_key=True) class FooView(ModelBase): foo = db.Column(db.Text, nullable=False) [??] class BarView(ModelBase): bar = db.Column(db.Text, nullable=False) [??] class BarbazView(BarView): baz = db.Column(db.Text, nullable=False) [??] class MyLargeModel(FooView, BarbazView): [??]  Where each of the view class retain as much of the SQLA model properties as possible (loading, and ideally persisting). In a way, something similar to what SQLA-Marshmallow etc do for schemas (except while keeping an actual ORM model). Is there any known pattern that could help me here? [Edit] After further digging, it seems like SQLAlchemy's Single table inheritance might hold the key to what we are trying to do, but it seems to require a polymorphic_on column to split vertically on (I only want to split horizontally).

Answer: After a lot of experimenting, I finally found a solution that, while not extremely satisfying (it relies on a fairly ugly __table__ definition) seems to fit my bill exactly. Sharing for any future searches: from redatabase.database import db from redatabase.models import ValuationRequest from sqlalchemy.orm.relationships import RelationshipProperty class ModelViewMeta(type(db.Model)): def __new__(mcs, name, bases, attrs): parent_model = attrs.get('_' + name + '__parent_model', None) included_attributes = attrs.get('_' + name + '__included_attributes', []) if parent_model: attrs['__table__'] = parent_model.__table__.select().with_only_columns([ getattr(parent_model, attribute) for attribute in included_attributes if not isinstance(getattr(parent_model, attribute).property, RelationshipProperty) ]).subquery() for attribute in included_attributes: attrs[attribute] = getattr(parent_model, attribute) else: attrs['__abstract__'] = True return super().__new__(mcs, name, bases, attrs) class ModelView(db.Model, metaclass=ModelViewMeta): def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) class FooModelView(ModelView): __parent_model = OriginalModel __included_attributes = ['id', 'foo']  The class FooModelView above will correctly expose only __included_attributes while still keeping all the properties expected from a SQLAlchemy model.
Question: What I want to achieve To scrape an website using AWS Lambda and save the data on S3. The issues I'm having When I execute Lambda, the following error message appears.  { "errorMessage": "Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "fb66bea9-cbad-4bd3-bd4d-6125454e21be", "stackTrace": [] }  Code The minimum Lambda code is as follows. import requests import boto3 def lambda_handler(event, context): s3 = boto3.client('s3') upload_res = s3.put_object(Bucket='horserace-dx', Key='/raw/a.html', Body='testtext') return event  An layer was added to the Lambda. Files were save in python folder using the commands below , frozen in a zip file, then uploaded to AWS Lambda as a layer. !pip install requests -t ./python --no-user !pip install pandas -t ./python --no-user !pip install beautifulsoup4 -t ./python --no-user   The bucket horserace-dx exists The folder raw exists The role of the Lambda is properly set. It can read from and write to S3 The runtime of the Lambda is Python 3.9. The python version of the local computer is 3.9.13.  What I did so far I google "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" and found some suggestions. I made the layer with the following code and tried again in vain. !pip install requests -t ./python --no-user !pip install pandas -t ./python --no-user !pip install beautifulsoup4 -t ./python --no-user !pip install urllib3==1.26.15 -t ./python --no-user  So what should I do to achieve what I want to achieve? Any suggestions would be greatly appreciated.

Answer:  Execute the following commands. pip install requests==2.25.0 -t ./python --no-user pip install beautifulsoup4 -t ./python --no-user pip install pytz -t ./python --no-user  On PyPI, download the following whl files from the pages of numpy and pandas    numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl   Unzip the files and move the contents to the python folder.  Zip the python folder and upload it to AWS Lambda Layer.  Set the layer to the Lambda.  Then the code runs without errors.