eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was trying to embed some documents on postgresql with the help of pgvector extension and langchain. Unfortunately I'm having trouble with the following error: (psycopg2.errors.UndefinedObject) type "vector" does not exist LINE 4: embedding VECTOR(1536), ^ [SQL: CREATE TABLE langchain_pg_embedding ( collection_id UUID, embedding VECTOR(1536), document VARCHAR, cmetadata JSON, custom_id VARCHAR, uuid UUID NOT NULL, PRIMARY KEY (uuid), FOREIGN KEY(collection_id) REFERENCES langchain_pg_collection (uuid) ON DELETE CASCADE ) ]   My environment info:  pgvector docker image ankane/pgvector:v0.4.1 python 3.10.6, psycopg2 2.9.6, pgvector 0.1.6   List of installed extensions on postgres  Name | Version | Schema | Description ---------+---------+------------+-------------------------------------------- plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language vector | 0.4.1 | public | vector data type and ivfflat access method  I've tried the following ways to resolve:  Fresh installing the Postgres docker image with pgvector extension enabled. Manually install the extension with the official instruction. Manually install the extension on Postgres like the following:  CREATE EXTENSION IF NOT EXISTS vector SCHEMA public VERSION "0.4.1";  But no luck.

Answer: Update 17th July 2023 As previously I mentioned my issue was somewhere else in my configuration, here is the other reason that may be responsible for the error,  The pgvector extension isn't enabled in the database you are using. Make sure you run CREATE EXTENSION vector; in each database you are using for storing vectors. The vector schema is not in the search_path. Run SHOW search_path; to see the available schemas in the search path and \dx to see the list of installed extensions with schemas.   Unfortunately, the issue was somewhere else. My extension installation and search_path schema were totally okay for the defined database I was supposed to use. But my environment variable which was responsible for which database to use, got messed up and was using the default database postgres instead of my defined database, which didn't have the extension enabled.
Question: I'm working with langchain and ChromaDb using python. Now, I know how to use document loaders. For instance, the below loads a bunch of documents into ChromaDb: from langchain.embeddings.openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings() from langchain.vectorstores import Chroma db = Chroma.from_documents(docs, embeddings, persist_directory='db') db.persist()  But what if I wanted to add a single document at a time? More specifically, I want to check if a document exists before I add it. This is so I don't keep adding duplicates. If a document does not exist, only then do I want to get embeddings and add it. How do I do this using langchain? I think I mostly understand langchain but have no idea how to do seemingly basic tasks like this.

Answer: Filter based solely on the Document's Content Here is an alternative filtering mechanism that uses a nice list comprehension trick that exploits the truthy evaluation associated with the or operator in Python: # Create a list of unique ids for each document based on the content ids = [str(uuid.uuid5(uuid.NAMESPACE_DNS, doc.page_content)) for doc in docs] unique_ids = list(set(ids)) # Ensure that only docs that correspond to unique ids are kept and that only one of the duplicate ids is kept seen_ids = set() unique_docs = [doc for doc, id in zip(docs, ids) if id not in seen_ids and (seen_ids.add(id) or True)] # Add the unique documents to your database db = Chroma.from_documents(unique_docs, embeddings, ids=unique_ids, persist_directory='db')  In the first line, a unique UUID is generated for each document by using the uuid.uuid5() function, which creates a UUID using the SHA-1 hash of a namespace identifier and a name string (in this case, the content of the document). The if condition in the list comprehension checks whether the ID of the current document exists in the seen_ids set:  If it doesn't exist, this implies the document is unique. It gets added to seen_ids using seen_ids.add(id), and the document gets included in unique_docs. If it does exist, the document is a duplicate and gets ignored.  The or True at the end is necessary to always return a truthy value to the if condition, because seen_ids.add(id) returns None (which is falsy) even when an element is successfully added. This approach is more practical than generating IDs using URLs or other document metadata, as it directly prevents the addition of duplicate documents based on content rather than relying on metadata or manual checks.
Question: I got an error says ValueError: `run` not supported when there is not exactly one output key. Got ['answer', 'sources', 'source_documents'].  Here's the traceback error File "C:\Users\Science-01\anaconda3\envs\gpt-dev\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "C:\Users\Science-01\Documents\Working Folder\Chat Bot\Streamlit\alpha-test.py", line 67, in <module> response = chain.run(prompt, return_only_outputs=True) File "C:\Users\Science-01\anaconda3\envs\gpt-dev\lib\site-packages\langchain\chains\base.py", line 228, in run raise ValueError(  I tried to run langchain on Streamlit. I use RetrievalQAWithSourcesChain and ChatPromptTemplate Here is my code import os import streamlit as st from apikey import apikey from langchain.document_loaders import PyPDFLoader from langchain.document_loaders import DirectoryLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import Chroma from langchain.chains import RetrievalQAWithSourcesChain from langchain.llms import OpenAI from langchain.prompts.chat import ( ChatPromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate, ) from langchain.chat_models import ChatOpenAI os.environ['OPENAI_API_KEY'] = apikey st.title('🐔 OpenAI Testing') prompt = st.text_input('Put your prompt here') loader = DirectoryLoader('./',glob='./*.pdf', loader_cls=PyPDFLoader) pages = loader.load_and_split() text_splitter = RecursiveCharacterTextSplitter( chunk_size = 1000, chunk_overlap = 200, length_function = len, ) docs = text_splitter.split_documents(pages) embeddings = OpenAIEmbeddings() docsearch = Chroma.from_documents(docs, embeddings) system_template = """ Use the following pieces of context to answer the users question. If you don't know the answer, just say that "I don't know", don't try to make up an answer. ---------------- {summaries}""" messages = [ SystemMessagePromptTemplate.from_template(system_template), HumanMessagePromptTemplate.from_template("{question}") ] prompt = ChatPromptTemplate.from_messages(messages) chain_type_kwargs = {"prompt": prompt} llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0, max_tokens=256) # Modify model_name if you have access to GPT-4 chain = RetrievalQAWithSourcesChain.from_chain_type( llm=llm, chain_type="stuff", retriever=docsearch.as_retriever(search_kwargs={'k':2}), return_source_documents=True, chain_type_kwargs=chain_type_kwargs ) if prompt: response = chain.run(prompt, return_only_outputs=True) st.write(response)  It seems like the error is in chain.run(), anyone know how to solve this error?

Answer: I found the solution, change this code if prompt: response = chain.run(prompt, return_only_outputs=True) st.write(response)  to this if st.button('Generate'): if prompt: with st.spinner('Generating response...'): response = chain({"question": prompt}, return_only_outputs=True) answer = response['answer'] st.write(answer) else: st.warning('Please enter your prompt')  I also added st.button, st.spinner, and st.warning (optional)
Question: I'm creating a C++ extension for python. It creates a module parent that contains a sub-module child. The child has one method hello(). It works fine if I call it as import parent parent.child.hello() > 'Hi, World!'  If I try to import my function it fails import parent from parent.child import hello > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ModuleNotFoundError: No module named 'parent.child'; 'parent' is not a package parent.child > <module 'child'>  here is my code setup.py from setuptools import Extension, setup # Define the extension module extension_mod = Extension('parent', sources=['custom.cc']) # Define the setup parameters setup(name='parent', version='1.0', description='A C++ extension module for Python.', ext_modules=[extension_mod], )  and my custom.cc #include <Python.h> #include <string> std::string hello() { return "Hi, World!"; } static PyObject* hello_world(PyObject* self, PyObject* args) { return PyUnicode_FromString(hello().c_str()); } static PyMethodDef ParentMethods[] = { {nullptr, nullptr, 0, nullptr} }; static PyMethodDef ChildMethods[] = { {"hello", hello_world, METH_NOARGS, ""}, {nullptr, nullptr, 0, nullptr} }; static PyModuleDef ChildModule = { PyModuleDef_HEAD_INIT, "child", "A submodule of the parent module.", -1, ChildMethods, nullptr, nullptr, nullptr, nullptr }; static PyModuleDef ParentModule = { PyModuleDef_HEAD_INIT, "parent", "A C++ extension module for Python.", -1, ParentMethods, nullptr, nullptr, nullptr, nullptr }; PyMODINIT_FUNC PyInit_parent(void) { PyObject* parent_module = PyModule_Create(&ParentModule); if (!parent_module) { return nullptr; } PyObject* child_module = PyModule_Create(&ChildModule); if (!child_module) { Py_DECREF(parent_module); return nullptr; } PyModule_AddObject(parent_module, "child", child_module); return parent_module; }  I install and build with python setup.py build install. So, how do I make sure that my parent is a package? My code is a toy example but I actually want both modules defined on C++ level. I don't want to split them into several modules - since they are sharing some C++ code. I'm hoping for something similar to approach of this answer Python extension with multiple modules

Answer: Doing this from within the extension is a "simple" matter of emulating the behavior for modules that the import system recognizes as packages. (Depending on context, it might be nicer to provide an import hook that did the same thing from the outside.) Just a few changes are needed:  Make the name of the child "parent.child". Make parent a package: PyModule_AddObject(parent_module, "__path__", PyList_New(0)); PyModule_AddStringConstant(parent_module, "__package__", "parent");   Make child a member of that package: PyModule_AddStringConstant(child_module, "__package__", "parent");   Update sys.modules as Python would if it had performed the import: PyDict_SetItemString(PyImport_GetModuleDict(), "parent.child", child_module);    Of course, several of these calls can fail; note that if PyModule_AddObject fails you have to drop the reference to the object being added (which I very much did not do here for clarity).
Question: I have a program with an expander next to a button, but the button is smaller than the expander, and it bothers me a little. Is it possible to make the button bigger/make the expander's height smaller in only python. I have found solutions online using css, but I am just using python for my code. Here is my code if anyone wants to look at it:  instructionCol, buttonCol = st.columns([4,1]) with instructionCol: with st.expander("Instructions"): st.write("Pretend these are the instructions.") with buttonCol: st.button("\nRestart\n", on_click=board.reset)  Here is also what it looks like: 

Answer: You can use st.markdown(css, unsafe_allow_html=True) directly inside the Python code: import streamlit as st st.markdown( """ <style> button { height: auto; padding-top: 10px !important; padding-bottom: 10px !important; } </style> """, unsafe_allow_html=True, ) instructionCol, buttonCol = st.columns([4,1]) with instructionCol: with st.expander("Instructions"): st.write("Pretend these are the instructions.") with buttonCol: st.button("\nRestart\n", on_click=board.reset)  It gives:  Notice that the button is now aligned with the expander (because I made it a bit bigger). Since you have found css solutions already, you can replace the css I generated with the one you found.
Question: Solved According to this: https://stackoverflow.com/a/57886655/15537469 and to this: https://stackoverflow.com/a/74918400/15537469 I make a Multi-stage Docker build with Poetry and venv FROM python:3.10-buster as py-build RUN apt-get update && apt-get install -y \ build-essential \ curl \ software-properties-common \ && rm -rf /var/lib/apt/lists/* RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry python3 - COPY . /app WORKDIR /app ENV PATH=/opt/poetry/bin:$PATH RUN poetry config virtualenvs.in-project true && poetry install FROM python:3.10-slim-buster EXPOSE 8501 HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health COPY --from=py-build /app /app WORKDIR /app CMD ./.venv/bin/python ENTRYPOINT ["streamlit", "run", "mtcc/app.py", "--server.port=8501", "--server.address=0.0.0.0"]  My build is going well but when I run my docker container I get this error: docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "streamlit": executable file not found in $PATH: unknown. I don't know if this is really useful but here is my file structure mtcc ├── mtcc │ └── app.py └── Dockerfile 

Answer: With poetry config virtualenvs.in-project true poetry will create a virtual environment in the .venv directory and install all it's dependencies in it. The typical approach is to activate a virtual environment before using it. Typically this is done with the .venv/bin/activate (or with poetry run / poetry shell). E.g. you could do something like: CMD source /app/.venv/bin/activate && exec streamlit run ...  Alternatively you can also activate a virtual environment by manipulating the path environment variable. If you prepend the path to the bin directory of the virtual env, the Docker environment will find all binaries: ENV VIRTUAL_ENV=/app/.venv ENV PATH="$VIRTUAL_ENV/bin:$PATH" CMD ["streamlit", "run", ...]  You can find these methods, and extended explanations at https://pythonspeed.com/articles/activate-virtualenv-dockerfile/.
Question: I am trying to upgrade pip by doing: pip install --upgrade pip  And I got: Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: pip in /home/VICOMTECH/bdacosta/.local/lib/python3.8/site-packages (22.0.4) Collecting pip Using cached pip-23.1.2-py3-none-any.whl (2.1 MB) WARNING: No metadata found in /home/mypersonal/path/.local/lib/python3.8/site-packages WARNING: Error parsing requirements for pip: [Errno 2] No such file or directory: '/home/mypersonal/path/.local/lib/python3.8/site-packages/pip-22.0.4.dist-info/METADATA' Installing collected packages: pip Attempting uninstall: pip WARNING: No metadata found in /home/mypersonal/path/.local/lib/python3.8/site-packages Found existing installation: pip 22.0.4 ERROR: Cannot uninstall pip 22.0.4, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps pip==22.0.4'.  I've tried to solve as shown in this similar question but I encountered the same error: pip install --force-reinstall --no-deps pip  I still get: Defaulting to user installation because normal site-packages is not writeable Collecting pip Using cached pip-23.1.2-py3-none-any.whl (2.1 MB) WARNING: No metadata found in /home/mypersonal/path/.local/lib/python3.8/site-packages Installing collected packages: pip Attempting uninstall: pip WARNING: No metadata found in /home/mypersonal/path/.local/lib/python3.8/site-packages Found existing installation: pip 22.0.4 ERROR: Cannot uninstall pip 22.0.4, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps pip==22.0.4'.  And if I go to the specified folder by doing: cd /home/mypersonal/path/.local/lib/python3.8/site-packages/pip-22.0.4.dist-info && ls  I just find: REQUESTED  This problem also does not allow me to install or upgrade any new package. Any suggestion about what I could do?

Answer: The problem was due because there were two versions of pip in the site-packages. In the folder /home/mypersonal/path/.local/lib/python3.8/site-packages/ weirdly there were:  pip-23.1.2.dist-info (correct and up-to-date) pip-22.0.4.dist-info (root of the problem).  I just removed the second one and everything started to work well again, including installing new packages.
Question: Following documentation, and the code snippet provided from https://docs.sqlalchemy.org/en/14/core/connections.html#streaming-with-a-fixed-buffer-via-yield-per (posted directly below), my query is not being batched into 50_000 rows. with engine.connect() as conn: result = conn.execution_options(yield_per=100).execute(text("select * from table")) for partition in result.partitions(): # partition is an iterable that will be at most 100 items for row in partition: print(f"{row}")  Here I am writing the results of a SQL query to a CSV file: with engine.connect() as conn: # connect to database with open(csv_path, mode='w', newline='') as f: c = csv.writer(f, quoting=csv.QUOTE_MINIMAL) c.writerow(columns) # write column headers result = conn.execution_options(yield_per=50_000).execute(sql_string) # execute query that results in 110k rows, expecting batches of 50k rows for partition in result.partitions(): for row in tqdm.tqdm(partition): # each partition only has 1 row, when I expect 50k rows c.writerow(row)  This sql_string results in 110k rows, so I expect 3 iterations of result.partitions(), however, I am seeing 110k iterations. I fear this is equivalent to DDOS-ing my own SQL Server database. I've also tried doing this without partitions(), and the same thing happens - no batching.  with engine.connect() as conn: with open(csv_path, mode='w', newline='') as f: c = csv.writer(f, quoting=csv.QUOTE_MINIMAL) c.writerow(columns) for row in tqdm.tqdm(conn.execution_options(yield_per=50_000).execute(sql_string)): # I expect 3 iterations, but I get 110k. c.writerow(row)  The server seems to handle this ok, I get 22k iterations / second, but I still wonder, am I DDOS-ing myself? I am using SQL Alchemy 1.4.42 (pip install --upgrade 'sqlalchemy==1.4.42') and Microsoft SQL Server 2019 - 15.0.X (X64)

Answer: The yield_per argument was having no effect in execute_options. In the example query snippets I posted in my question, fetchone() gets called N times, where N is the query result's row count. I discovered these fetchone() calls by getting the traceback from doing ctrl-C during an execution. That's a lot of server calls. Instead of concerning myself with yield_per, I was able to achieve pagination by using fetchmany(batch_size).  with engine.connect() as conn: print('Connected!...') with open(csv_path, mode='w', newline='', encoding='utf-8') as f: c = csv.writer(f, quoting=csv.QUOTE_MINIMAL) c.writerow(columns) result = conn.execute(sql_string) counter = 1 while True: print(f'Querying batch {counter}...') rows = result.fetchmany(50_000) # batch size print(f'Contains {len(rows)} rows...') if not rows: break print(f'Writing batch {counter} to CSV...') for row in rows: c.writerow(row) counter += 1 result.close()  I don't understand the reason for yield_per in SQLAlchemy 1.4.42, or why the the SQLAlchemy docs don't reference using fetchmany() instead. Oh well.
Question: I am working on a python application based on PyQt5. Everything was running good until I redo my PC and reinstall windows again because of some issue. I had copied my environment and after reinstalling Anaconda, I copied that environment again in env folder. Now the problem is that, when I run my code in PyCharm IDE, it displays and error dialog like this:  I have tried multiple solutions such as: Solution 1: Change the QT Files Location Sometimes, a simple trick such as changing the QT files location is enough to get rid of the error. Here’s how you can do it:  Launch File Explorer and open This PC. Using the Search field, search for pyqt5_tools. When Windows finishes the search, right-click the pyqt5_tools and head to Open folder location. Head to PyQt5 > Qt > bin. Copy the platforms folder. in my case bin is not available in this folder Make a new search for site-packages and open the folder. There, paste the platforms folder. Windows will warn you there’s already a folder with the same name. Click Replace the files in the destination.  Solution 2: Run an SFC Scan There’s a chance Windows display the “Application failed because no QT platform plugin could be initialized” error due to corrupt system files. Fortunately, Windows has a built-in tool to help you fix the problem. In the Start menu search bar, search for command prompt and select Run as administrator. Then, run the sfc /scannow command line. Windows will scan and automatically replace any corrupted system file. Solutions Reference Nothing worked for me, I'm getting the same error all the time. How can I resolve the error displayed in the figure above? Any help will be appreciated, Thanks.

Answer: Updated Answer to this question Answer Reference To add PyQt5's library path to your PATH environment variable: 1. Open the Edit Environment Variables dialog. 2. Select the appropriate Path variable (either User or System variables). (Note: I have changed in system variables) 3. Edit the variable and add the PyQt5 library path as three entries at the top. 4. Add the following 3 entries, and make sure the entries are in the desired order and at the top of path variable. 5. Save the changes. C:\ProgramData\Anaconda3\envs\ann_tool\Lib\site-packages\PyQt5\Qt5\bin C:\ProgramData\Anaconda3\envs\ann_tool\Lib\site-packages\PyQt5\Qt5\plugins C:\ProgramData\Anaconda3\envs\ann_tool\Lib\site-packages\PyQt5\Qt5\plugins\platforms  6. Also, if you have a variable named: QT_PLUGIN_PATH, you want to put this line at the top of the list: C:\ProgramData\Anaconda3\envs\ann_tool\Lib\site-packages\PyQt5\Qt5\plugins  (Note: In my case, Anaconda is installed for all users so, Anaconda will be found in C:\ProgramData) 7. Once you have saved the changes to your environment variables, it is necessary to Restart PyCharm in order for it to recognize and locate the updated libraries..
Question: I am trying to convert a Pandas dataframe column data['hexValues'] consisting of hex values to decimal. data["decValues"] = data.apply(lambda row: int(str(row["hexValues"]), 16), axis=1)  This works but the column hexValues look like this 0 FF 1 F 2 nan 3 nan 4 FFFF 5 FFFF 6 F 7 F 8 F 9 F 10 FF 11 nan 12 nan  I want to skip nan values, so if the value is nan, no conversion will take place. I thought of using try-except-else but I am not sure how I can do it. Can you help me?

Answer: If nans are missing values use: data["decValues"] = [int(x, 16) if pd.notna(x) else np.nan for x in data["hexValues"]]  More general solution with try-except: def test(x): try: return int(x, 16) except ValueError: return np.nan data["decValues"] = data["hexValues"].apply(test)
Question: I would like to have an existing class A inherit B only if a runtime flag is turned on. Is this possible? Usually for these cases, I either  Just have A inherit B by default, and not use B if that flag is False  Create another class A_mod, that inherits from A and B, and use this class when that flag is true.  

Answer: You can use a conditional expression when specifying the parent class class A(B if flag else object):