eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm having an issue with circular imports in SQLAlchemy. I have two files foo.py and bar.py. foo.py defines a SQLAlchemy class Foo, and bar.py defines a class Bar. Both Foo and Bar are each other's foreign keys, so I map them to each other with Mapped["..."] to get type safety, however that means I need to import the actual classes aswell. This is causing a circular import error. What's the best way to handle this issue? What are some general best practices for dealing with circular imports in SQLAlchemy? Can't you have type safety in this case if you use a relationship bidirectionally? # foo.py from sqlalchemy import Column, Integer, ForeignKey from sqlalchemy.orm import relationship from .bar import Bar class Foo(Base): __tablename__ = 'foo' id = Column(Integer, primary_key=True) bar_id = Column(Integer, ForeignKey('bar.id')) bar: Mapped["Bar"] = relationship('Bar') # bar.py from sqlalchemy import Column, Integer, ForeignKey from sqlalchemy.orm import relationship from .foo import Foo class Bar(Base): __tablename__ = 'bar' id = Column(Integer, primary_key=True) foo_id = Column(Integer, ForeignKey('foo.id')) foo: Mapped["Foo"] = relationship('Foo')  Edit: Note that I can't remove the Bar and Foo imports because then Mapped["..."] will raise an undefined error for "..."

Answer: you can use TYPE_CHECKING. This constant is False at runtime, but True when mypy (and other type-checking tools) evaluate your code from __future__ import annotations from typing import TYPE_CHECKING if TYPE_CHECKING: from .bar import Bar else: Bar = "Bar" # foo.py from sqlalchemy import Column, Integer, ForeignKey from sqlalchemy.orm import relationship class Foo(Base): __tablename__ = 'foo' id = Column(Integer, primary_key=True) bar_id = Column(Integer, ForeignKey('bar.id')) bar: Mapped[Bar] = relationship('Bar')
Question: Let's say I have the following dataframe:     Code Price     AA1 10   AA1 20   BB2 30     And I want to perform the following operation on it: df.groupby("code").aggregate({ "price": "sum" })  I have tried playing with the new pyarrow dtypes introduced in Pandas 2.0 and I created 3 copies, and for each copy I measured execution time (average of 5 executions) of the operation above.     Code column dtype Price column dtype Execution time     Object float64 2.94 s   string[pyarrow] double[pyarrow] 49.5 s   string[pyarrow] float64 1.11 s     Can anyone explain why applying an aggregate function on a column with double pyarrow dtype is so slow compared to the standard numpy float64 dtype?

Answer: https://github.com/pandas-dev/pandas/issues/52070 Looks like groupby for arrow isn't implemented yet - so there's likely a arrow -> numpy happening internally leading to a loss of performance.
Question: I installed rasa on a virtual environment on windows. But while I am trying to check either rasa is installed or not, it is showing an error that says- ImportError: cannot import name 'CLOSED' from 'websockets.connection' I have reinstalled rasa, and installed websockets. But still getting the error. Python version is 3.10.2 Can anyone help me to solve this problem?

Answer: Try installing websockets version 10.0 like this: pip install websockets==10.0 It should help (it helped me)
Question: I have issues with jupyter extensions on ArchLinux. In particular, I get the following error: [W 2023-04-01 18:34:36.504 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_nbextensions_configurator. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. [W 2023-04-01 18:34:36.493 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.  How can I get rid of this error/warning? I tried removing the functions with Pip, but it did not work. Any ideas?

Answer: This is not an error, but a warning aimed at developers of notebook extensions: jupyter_nbextensions_configurator and notebook_shim respectively, not at a user like you. You do not need to do anything. It's worth pointing out that the next version of Jupyter Notebook (v7) will not require jupyter_nbextensions_configurator anymore as it will come with new extension manager (and a completely new extension ecosystem shared with JupyterLab), so there is very little incentive to address this deprecation warning in jupyter_nbextensions_configurator as it will cease to be relevant soon.
Question: This is my code: from fastapi import FastAPI, Depends, Security from fastapi.security import HTTPBearer bearer = HTTPBearer() @app.get("/") async def root(q = Security(bearer)): return {'q': q} @app.get("/Depends") async def root(q = Depends(bearer)): return {'q': q,}  Both routes give precisely the same result and act in the same manner. I checked the source code and found that the Security class inherits from the Depedends class. But I have no understanding in what way. Can you please show me the differences and why would I prefer to use Security over Depends.

Answer: TL;DR Use Security for security related dependencies as it is thought as a convenience for the devs. Use Depends when more general dependencies are needed. Nevertheless, you may use these two interchangeably (for security related requirements). Original answer The Security class is a more specialized version of Depends. In the source code https://github.com/tiangolo/fastapi/blob/master/fastapi/param_functions.py other types of dependencies are defined, such as Form, Body and File. As their names suggest, they are specialized for a particular task/data. Basically there is no big difference between the two. The Security is more specialized than Depends, so the output is the same. It is more of a convenience for the developer.
Question: I like the convenience of the following syntax to make maps: import geopandas world = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres')) world.plot(column='continent',legend=True,legend_kwds = {'loc':'lower left'})  However, I am looking for a way to change the shape of markers that this produces. By default (with a non-continuous variable), geopandas (or matplotlib) produces filled in circles. I would really like these markers to be filled in rectangles with a thin black border. I have tried using the marker parameter in plot to no success. I have also tried various arguments in legend_kwds.

Answer: Using geopandas makes it difficult to find a way to edit the legend's properties or get hold of the legend object. Here is a way to go. First, get the axis from the plot statement: ax = world.plot( ... )  the axis provides access to all the plot components and many functions to manipulate your plot, including the legend. Once you get hold of the legend, you can change its properties as needed. In the code below, the shape of the marker, edge color, edge width are set as required. import geopandas world = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres')) ax = world.plot(figsize=(12,9) ,column='continent',legend=True,legend_kwds = {'loc':'lower left'}) leg1 = ax.get_legend() for ea in leg1.legendHandles: ea.set_marker('s') ea.set_markeredgecolor('k') ea.set_markeredgewidth(0.5)   As far as I know, the option legend_kwds in the plot statement does not include the item to set the legend marker's shape.
Question: I'm trying to unpivot my data using pd.melt but no success so far. Each row is a business, and the data contains information about the business and multiple reviews. I want my data to have every review as a row. My first 150 columns are in groups of 15, each group column name shares the same pattern reviews/n/ for 0 < n < 9. (reviews/0/text, reviews/0/date, ... , reviews/9/date). The next 65 columns in the dataframe include more data about the business (e.g. business_id, address) that should remain as id_variables. My current data looks like this:     business_id address reviews/0/date reviews/0/text reviews/1/date reviews/1/text     12345 01 street 1/1/1990 "abc" 2/2/1995 "def"     and my new dataframe should have every review as a row instead of every business, and look like this:     business_id address review_number review_date review_text     12345 01 street 0 1/1/1990 "abc"   12345 01 street 1 2/2/1995 "def"     I tried using pd.melt but could not succeed in making code that produced something valuable to me.

Answer: You can use pandas.wide_to_long() to do what you want. However, you will need to rename your columns from the pattern reviews/N/COL to reviews/COL/N (or something similar) first, as wide_to_long() can only unpivot based on prefixes, whereas in your column names, you have a prefix and a suffix. You could do this manually or e.g. using the re module and an appropriate regex: df = df.rename(columns=lambda x: re.sub('reviews/(\d)/(.*)', r'review_\2\1', x))  After that, your data should look like this (note the changed colnames):     business_id address review_date0 review_text0 review_date1 review_text1     12345 01 street 1/1/1990 abc 2/2/1995 def     Now you can use pandas.wide_to_long() and use the stubnames parameter to specify the prefix of the columns that should be grouped when you unpivot. df = pd.wide_to_long(df, stubnames=['review_date','review_text'], i=['business_id', 'address'], j='review_number')  Finally, call .reset_index() to achieve the result you asked for. Full example: import re import pandas as pd df = pd.DataFrame({'business_id': 12345, 'address': '01 street', 'reviews/0/date': '1/1/1990', 'reviews/0/text': 'abc', 'reviews/1/date': '2/2/1995', 'reviews/1/text': 'def'}, index = [0]) df = df.rename(columns=lambda x: re.sub('reviews/(\d)/(.*)', r'review_\2\1', x)) df = pd.wide_to_long(df, stubnames=['review_date','review_text'], i=['business_id', 'address'], j='review_number').reset_index()  Result:     business_id address review_number review_date review_text     12345 01 street 0 1/1/1990 abc   12345 01 street 1 2/2/1995 def
Question: I have a csv file with the following structure: time,magnitude 0,13517 292.5669,370 620.8469,528 0,377 832.3269,50187 5633.9419,3088 20795.0950,2922 21395.6879,2498 21768.2139,647 21881.2049,194 0,3566 292.5669,370 504.1510,712 1639.4800,287 46709.1749,365 46803.4400,500  I'd like to split this csv file into separate csv files, like the following: File 1: time,magnitude 0,13517 292.5669,370 620.8469,528  File 2: time,magnitude 0,377 832.3269,50187 5633.9419,3088 20795.0950,2922 21395.6879,2498  and so on.. I've read several similar posts (e.g., this, this, or this one), but they all search for specific values in a column and save each groups of values into a separate file. However, in my case, the values of time column are not the same. I'd like to split base on a condition: If time = 0, save that row and all subsequent rows in a new file until the next time =0. Can someone please let me know how to do this?

Answer: With pandas, you can use groupby and boolean indexing : #pip install pandas import pandas as pd df = pd.read_csv("input_file.csv", sep=",") # <- change the sep if needed for n, g in df.groupby(df["time"].eq(0).cumsum()): g.to_csv(f"file_{n}.csv", index=False, sep=",")  Output :  time magnitude # <- file_1.csv 0.0000 13517 292.5669 370 620.8469 528 time magnitude # <- file_2.csv 0.0000 377 832.3269 50187 5633.9419 3088 20795.0950 2922 21395.6879 2498
Question: I have a pandas dataframe with the following format:     UserId Date BookId     1 2022-07-15 10   1 2022-07-16 11   1 2022-07-16 12   1 2022-07-17 12     From this table, what I want to obtain is the number of new BookId on each consecutive day for each user. For example, based on the table above, the user read two new books on 2022-07-16, and did not read a new book on 2022-07-17 since s/he read it already on the previous day. Here is the expected outcome:     UserId 2022-07-16 2022-07-17     1 2 0     I feel like this task could be done by grouping data by UserId and Date, and then using the apply lambda function. However, I could not manage it. I ended up with the following code, which uses the for loop. Is there a way to achieve this without a loop with a shorter code? df = studentAnswers.groupby('StudentId') df.apply(findObjDiff) def findObjDiff(df): print(df.StudentId.head(3)) dataDict = {} dates = list(df.Date) dates.sort() for d in dates: ixNext = dates.index(d) + 1 if(ixNext > len(dates)): break dateNext = dates[ixNext] objListPrev = set(df[df.Date == d].ObjectiveId) objListNext = set(df[df.Date == dateNext].ObjectiveId) dataDict[df.StudentId] = {dateNext : {'Different': len(objListPrev - objListNext)}} return dataDict 

Answer: Using duplicated and a pivot_table: (df.assign(count=~df['BookId'].duplicated()) .pivot_table(index='UserId', columns='Date', values='count', aggfunc='sum') .astype(int).reset_index().rename_axis(columns=None) )  Considering only consecutive days for the duplicates: s = df.groupby(pd.to_datetime(df['Date']))['BookId'].agg(set) (df.assign(count=(s-s.shift(1, freq='D')).str.len().to_numpy()) .pivot_table(index='UserId', columns='Date', values='count', aggfunc='sum') .astype(int).reset_index().rename_axis(columns=None) )  Output:  UserId 2022-07-15 2022-07-16 2022-07-17 0 1 1 2 0
Question: I'm using isolated with poetry/venv Python: 3.11.2 with Ubuntu WSL2 22.04 python app.py  Gives me the following error: Traceback (most recent call last): File "/home/lara/projects/qtapp/app.py", line 4, in <module> from PySide6.QtGui import QGuiApplication ImportError: libEGL.so.1: cannot open shared object file: No such file or directory  app.py: from PySide6.QtGui import QGuiApplication from PySide6.QtQml import QQmlApplicationEngine app = QGuiApplication() engine = QQmlApplicationEngine() engine.load("myqmlfile.qml") app.exec()  pyproject.toml dependencies [tool.poetry.dependencies] python = ">=3.11,<3.12" pyside6 = "^6.4.3" [tool.poetry.group.dev.dependencies] pytest = "^7.2.2" pytest-cov = "^4.0.0" isort = "^5.12.0" black = "^23.3.0" poethepoet = "^0.19.0" 

Answer: You can search packages.ubuntu.com to figure out which package provides a certain file. In this case it's libegl1. Install it in WSL with: apt install libegl1
Question: On this page I read this:  Coroutines in the asyncio module are not limited by the Global Interpreter Lock or GIL.  But how is this possible if both the asyncio event loop and the threading threads are running in a single Python process with GIL? As far as I understand, the impact of the GIL on asyncio will not be as strong as on threading, because in the case of threading, the interpreter will switch to useless operations, such as time.sleep(). Therefore, when working with asyncio, it is recommended to use asyncio.sleep(). I understand that these tools are designed for slightly different things, threading is more often used to execute "legacy" blocking code for IO-Bound operations, and asyncio for non-blocking code.

Answer: There are really two kinds of answers to this, depending on whether you take the GIL as the concrete implementation or the conceptual limitation. The answer is No'ish for the former, and Yes'ish for the latter.  No, asyncio concurrency is not bound to the GIL. The GIL exists to synchronise thread concurrency: When more than one thread is active at once, only the thread that owns the GIL may execute Python code. This means that if another thread needs to run, ownership of the GIL must pass from the current executing thread to the other thread. This is realised via preemptive concurrency, meaning the currently executing thread is forcefully paused regularly. As such, the GIL is the mechanism handling the switching of control between the currently executing thread and all other threads. This is a very wasteful mechanism that becomes worse the more threads there are. In contrast, asyncio has its own concurrency synchronisation: async/await provides cooperative concurrency by design. That means the currently executing task can voluntarily yield control to any other waiting task. This allows the asyncio event loop to freely schedule tasks. As such, the GIL is not involved in switching control between the currently executing task and all other tasks. An async framewok such as asyncio can achieve very efficient task switching, regardless of the number of tasks.  Yes, asyncio is bound by the GIL. While asyncio can efficiently switch tasks, it cannot run more than one task at any moment. As such, it does not actually avoid the conceptual limitation of the GIL: running more than one operation at once. While asyncio applications avoid the inefficiency of the GIL for many operations, it does not avoid the limitation of the GIL for parallel operations. Notably, solving this limitation would require solving the exact same challenge as for removing the GIL: providing a safe synchronisation when executing Python code in parallel. In addition, asyncio still cannot execute blocking code concurrently as tasks but must fallback to threading for these. Thus, asyncio's backend for executing blocking code concurrently is directly bound by the GIL.
Question: I have a data frame with 3 columns: A, B and C as below: A: 1 2 3 B: 10 20 30 C: 100 200 300 I need to save the values only in a csv file in "one" row like this ( no need to save the column name) 1 2 3 10 20 30 100 200 300 I tried to create a new data frame with only one coumn using pandas, and transpose it, then write it to a csv file. But I couln't figure out how. Could you please help me?

Answer: You can use unstack, to_frame and T: (df.unstack().to_frame().T .to_csv('out.csv', header=None, index=None, sep=' ') )  Or, with help of numpy's ravel: (pd.DataFrame([df.to_numpy().ravel('F')]) .to_csv('out.csv', header=None, index=None, sep=' ') )  Output file: 1 2 3 10 20 30 100 200 300
Question: i have problem while building docker image on python : below execution process takes long time around 20 minutes: Building wheel for pandas (pyproject.toml): still running... Building wheel for pandas (pyproject.toml): still running... Building wheel for pandas (pyproject.toml): still running... Building wheel for pandas (pyproject.toml): still running... Building wheel for pandas (pyproject.toml): still running...  Dockerfile: FROM python:3.11.2-buster WORKDIR /app COPY . . RUN pip install -r /app/requirements.txt CMD ["uvicorn", "main:app", "--host=0.0.0.0", "--port=80"]  requirement.txt: fastapi uvicorn pydantic[dotenv] requests python-dotenv==0.19.2 pandas==1.4.3 numpy==1.24.2 scikit-learn==1.2.2 

Answer: There is no building wheel for Pandas==1.4.3 and Python==3.11 so the fallback is to download the source archive (pandas-1.4.3.tar.gz) and build it from scratch. You have 2 options:  Downgrade your python version from 3.11 to 3.10 in Dockerfile:  FROM python:3.10.10-buster   Upgrade the pandas version from 1.4.3 to 1.5.3 in requirements.txt:  pandas==1.5.3  In both cases (Python 3.10, Pandas 1.4) or (Python 3.11, Pandas 1.5), there is no compilation time with your requirements.
Question: I was wondering if there is a generic way to find out if your version of python is supported by a specific module? For example, let us say that I have python 3.11 installed on my computer and I want to install the modules biopython and lru-dict. Going to their respective pypi entries biopython shows this in their Project description:  Python Requirements We currently recommend using Python 3.10 from http://www.python.org Biopython is currently supported and tested on the following Python implementations: Python 3.7, 3.8, 3.9, 3.10 and 3.11 – see http://www.python.org PyPy3.7 v7.3.5 – or later, see http://www.pypy.org  However, if we check the lru-dict entry, their Project description does not mention Python requirements. I did eventually find out that python 3.11 is not supported by finding a issue on their github page. Is there a simpler way of finding this information out?

Answer: No, there is no generic way. If the library has tests you can run them and check, but it might give you false positives as well as false negatives :shrug: The lru-dict has this fragment in their setup.py, which is one of the places you can check for supported versions:  'Programming Language :: C', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: Implementation :: CPython',  So as you can see, I don't think they knew about the issue.
Question: I am trying to drop duplicates, but based on some conditions. My dataframe looks like this: idx a b c d e f 1 1 ss1 0 25 A B 2 3 ss7 0 25 A B 3 5 ss5 0 12 C D 4 11 im3 0 12 C D 5 5 ss8 0 50 C K 6 9 im8 0 5 F G 7 8 ix6 0 5 F G  Rows are considered duplicates if the values of columns d, e and f together match other records in the dataframe subset=['d', 'e', 'f']. For example, rows 1 and 2 are duplicates, rows 3 and 4 are duplicates, and rows 6 and 7 are duplicates. The selection of which row to drop is based on column b.  If the value in column b begins with ss for both duplicates (rows 1 and 2), then anyone can be dropped  If one of the duplicates begins with ss and the other begins with a different format (rows 3 and 4), then the one that begins with ss should be kept.  If both duplicates in column b begin with anything other than ss (rows 6 and 7), then anyone can be selected.   Therefore, the expected output should be something like this: idx a b c d e f 2 3 ss7 0 25 A B 3 5 ss5 0 12 C D 5 5 ss8 0 50 C K 7 8 ix6 0 5 F G 

Answer: Sort by b key first (everything starts by 'ss' is moved to the end) then drop duplicates from ['d', 'e', 'f'] (keep the last): out = (df.sort_values('b', key=lambda x: x.str.startswith('ss')) .drop_duplicates(['d', 'e', 'f'], keep='last').sort_index()) # OR out = (df.sort_values('b', key=lambda x: x.str.startswith('ss')) .groupby(['d', 'e', 'f'], as_index=False).nth(-1).sort_index())  Output: >>> out idx a b c d e f 1 2 3 ss7 0 25 A B 2 3 5 ss5 0 12 C D 4 5 5 ss8 0 50 C K 6 7 8 ix6 0 5 F G
Question: I want to convert df into df_target, by keeping only the value in the last non-nan column (for each row individually). The other value should be set to nan. The following code already achieves what I want, but is very slow for large DataFrames. Is there any solution that is faster? import pandas as pd data = { 'A': [1, 2, 3, pd.NA, 5], 'B': [pd.NA, pd.NA, pd.NA, 4, 5], 'C': [pd.NA, pd.NA, 3, 4, pd.NA], } df = pd.DataFrame(data) data_target = { 'A': [1, 2, pd.NA, pd.NA,pd.NA], 'B': [pd.NA, pd.NA, pd.NA, pd.NA, 5], 'C': [pd.NA, pd.NA, 3, 4, pd.NA], } df_target = pd.DataFrame(data_target) df_out = df.apply(lambda row: row.where(row.index == row.last_valid_index(), pd.NA), axis=1) print(df_out.equals(df_target)) 

Answer: Do: df.where(df.notnull().iloc[:,::-1].cumsum(axis=1).le(1), pd.NA)  The idea is to count (cumsum) the non-NA (notnull) from right to left (.iloc[:,::-1]) on each row (axis=1), then mask everything with >=2 NA as NA Output:  A B C 0 1 <NA> <NA> 1 2 <NA> <NA> 2 <NA> <NA> 3 3 <NA> <NA> 4 4 <NA> 5 <NA>