eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: pip is recommended to be run under local user, not root. This means if you do sudo python ..., you will not have access to your Python libs installed via pip. How can I run Python (or a pip installed bin/ command) under root / sudo (when needed) while having access to my pip libraries?

Answer: So the issue you're likely having is that you have pip installed the packages you want to a user specific site-packages directory, which then isn't being included in the sys.path module search list generated by sites.py when Python starts up for the root user. There are a couple of workaround for this to add the additional paths to be searched for modules.  Use the PYTHONPATH environment variable. This should be set to semi-colon delimited and terminated list of paths. Directly modify sys.path at the start of your script before running any other imports. I often see this proposed on SO as a workaround around by people who haven't really grasped absolute vs relative imports.  I have an example here using both methods. I'm on Windows in a virtual environment, but the principles would be the same in Linux. Setup: (venv) ~/PythonVenv/playground311 (main) $ export PYTHONPATH="C:\Path\From\PythonPath\;E:\Second\PythonPath\Path\;"  Start of script: import sys # Insert at position 1 as position 0 is this script's directory. sys.path.insert(1, r"C:\Path\Inserted\By\Script") for path in sys.path: print(path)  Output c:\Users\nighanxiety\PythonVenv\playground311 C:\Path\Inserted\By\Script C:\Path\From\PythonPath E:\Second\PythonPath\Path C:\Users\nighanxiety\PythonVenv\playground311 C:\python311\python311.zip C:\python311\DLLs C:\python311\Lib C:\python311 C:\Users\nighanxiety\PythonVenv\playground311\venv C:\Users\nighanxiety\PythonVenv\playground311\venv\Lib\site-packages  If I weren't in a virtual environment, the last few entries are different: C:\Users\nighanxiety\PythonVenv\playground311 C:\Path\Inserted\By\Script C:\Path\From\PythonPath E:\Second\PythonPath\Path C:\Users\nighanxiety\PythonVenv\playground311 C:\Python311\python311.zip C:\Python311\Lib C:\Python311\DLLs C:\Python311 C:\Python311\Lib\site-packages  I strongly recommend trying a virtual environment and using pip to install the desired packages there first, as Sauron suggests in his answer. The root user should still correctly use the virtual environment paths as long as the environment is activated. See How does activating a python virtual environment modify sys.path for more info. If that doesn't work, then configuring PYTHONPATH with the correct absolute path to your site-packages should be a better option than hacking sys.path. Modifying sys.path makes sense if a specific override is needed for that specific script, but is otherwise a bit of a hack.
Question: I have a dataset: import pandas as pd data = [ ('A', 'X'), ('A', 'X'), ('A', 'Y'), ('A', 'Z'), ('B', 1), ('B', 1), ('B', 2), ('B', 2), ('B', 3), ('B', 3), ('C', 'L-7'), ('C', 'L-9'), ('C', 'L-9'), ('T', 2020), ('T', 2020), ('T', 2025) ] df = pd.DataFrame(data, columns=['ID', 'SEQ']) print(df)  I want to create a key grouping ID and SEQ in order to select the first 2 rows of each different SEQ within each ID Group For instance the ID A, has 3 distinct keys "A X", "A Y" and "A Z" in the order of the dataset the first two keys are "A X" and "A Y" thus I must select the first two rows (if available) of each thus "A X", "A X", "A Y" why? because "A Z" is another key. I've tried using the groupby and head functions, but I couldn't find a way to achieve this specific result. What can I try next? (df .groupby(['ID','SEQ']) .head(2) )  This code is returning the original dataset and I wonder if I can solve this problem using method chaining, as it is my preferred style in Pandas. The final correct output is: 

Answer: Your approach of using groupby and then head(2) is on the right track for getting the first 2 rows of each different SEQ within each ID group. However, the additional requirement is to get only the first 2 unique SEQ groups within each ID. To achieve this, you can: Create a new column that has the rank of unique SEQ within each ID group. Use this rank to filter out the data. Finally, use your original approach to get the first 2 rows of each SEQ within each ID group. Here's a solution using method chaining: result = (df .assign(rank=df.groupby('ID')['SEQ'].transform(lambda x: x.rank(method='dense'))) .query('rank <= 2') .groupby(['ID', 'SEQ']) .head(2) .drop(columns=['rank']) ) print(result)  This should give you the desired output.
Question: I want to create an snowflake stored procedure using snowpark. Inside of the procedure I will create some dataframe and performs some operations like filter, sort, join etc. At the end I just want to return that "dataframe", does snowpark supports that? Can someone provide an example of how to achieve this?

Answer: It is supported - Writing Snowpark Code in Python Worksheets  Python Worksheet    Run + Deploy    Code is wrapped as Stored Procedure and ready to be used    Call in SQL CALL ProcName();     Call in Python
Question: import numpy as np # Create two sample dataframes df1 = np.array([[0.000000,0.000000,0.000000], [0.090000,0.000000,0.000000], [0.190000,0.000000,0.000000], [0.280000,0.000000,0.000000], [0.380000,0.000000,0.000000], [0.470000,0.000000,0.000000], [0.570000,0.000000,0.000000], [0.660000,0.000000,0.000000], [0.760000,0.000000,0.000000], [0.850000,0.000000,0.000000]]) df2 = np.array([[0.470000,0.000000,0.000000], [0.570000,0.000000,0.000000], [0.660000,0.000000,0.000000], [0.760000,0.000000,0.000000], [0.850000,0.000000,0.000000] ]) df3 = np.where(np.isclose(df1[:, np.newaxis], df2))[0] print(df3)  I want to find the postion of df2 in df1 and the correct answer is [5, 6, 7, 8, 9] but the python output is [0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9] which does not make sense. How can I change the code to find the postion of df2 in df1 correctly?

Answer: If you only want to match the non-zeros, you can remove them: np.where(np.isclose(np.where(df1!=0, df1, np.nan)[:, None], df2))[0]  Output: array([5, 6, 7, 8, 9])  If you want a full match, including zeros (row per row), you can use: out = np.where(np.isclose(df1[:, None], df2).all(2))[0]  Output: array([5, 6, 7, 8, 9])
Question: I'm trying to use two lists of dictionaries to build an object, the dictionaries are built from a TAP connection to two different databases. Because of the data sources I cannot guarantee that any of the dictionaries will contain the information I need, so I've chosen a primary dictionary and if the information is not in there then I extract it from the second dictionary. Because I'm pulling the data from two different data sources, the field names from TAP differ for both sources, so I can't just do an intersection of the dictionaries. At the moment I can get it to work, but I'm not happy with the solution: for result in eeuresults: name=result['target_name'] axes=result['semi_major_axis'] period=result['period'] radius=result['radius'] if math.isnan(period): for r in eparesults: if r['pl_name'] == name: period=r['pl_orbper'] if math.isnan(axes): for r in eparesults: if r['pl_name'] == name: axes=r['pl_orbsmax'] if math.isnan(radius): for r in eparesults: if r['pl_name'] == name: radius=r['pl_radj']  I've tried using dictionary.get() to make it simpler, but it falls down if the value isn't in the second list of dictionaries. axes=result.get('semi_major_axis',[r['pl_orbper'] for r in eparesults if r['pl_name']==name][0])  Edit: Solution Chosen I ended up using rioV8's solution to reduce code duplication, it's not the most efficient solution, but it is elegant, readable and expandable; which is what I really wanted. Full function is below: def getsystemdata(name, epaname=None): def search_epa(name, value, key): if math.isnan(value): for r in eparesults: if r['pl_name'] == name: return r[key] return value if epaname == None: epaname=name service=pyvo.dal.TAPService("https://exoplanetarchive.ipac.caltech.edu/TAP") eparesults=service.search(f"select pl_name, pl_radj, pl_orbper, pl_orbsmax from pscomppars where hostname = '{epaname}' ") service=pyvo.dal.TAPService("http://voparis-tap-planeto.obspm.fr/tap") eeuresults=service.search(f"select target_name, radius, period, semi_major_axis from exoplanet.epn_core where star_name = '{name}'") planets=[] for result in eeuresults: name=result['target_name'] axes=search_epa(name, result['semi_major_axis'], 'pl_orbsmax') period=search_epa(name, result['period'], 'pl_orbper') radius=search_epa(name, result['radius'], 'pl_radj') planets.append(Planet(name.split(' ')[-1], axes, period, radius)) return planets 

Answer: remove code duplication def find_in_eparesults(name, value, epa_key): if math.isnan(value): for r in eparesults: if r['pl_name'] == name: value = r[epa_key] break return value for result in eeuresults: name=result['target_name'] axes=find_in_eparesults(name, result['semi_major_axis'], 'pl_orbper') period=find_in_eparesults(name, result['period'], 'pl_orbsmax') radius=find_in_eparesults(name, result['radius'], 'pl_radj')
Question: I analysed a triangle wave using the Fourier series by hand, and then using the scipy.fft package. What I'm getting are the same absolute values of the coefficients, but with opposite signs.  By hand: I took the interval [-1,1], calculated the integral to get a0 = 1, then a1, a3 and a5 using the result of integration:  And all the bn are 0. Then I simply constructed the series, where the 1st term is a0/2, the second a1×cos(n Pi t/T) and so on, and plotted these waves, which summed give a good approximation of the original signal. The coefficients are: a0 = 0.5 a1 = -0.4053 a3 = -0.0450 a5 = -0.0162    Scipy.fft: I defined the sampling frequency fs=50, and created the space for the function with space=np.linspace(-1,1,2*fs). Then defined the fft1 = fft.fft(triang(space)), where "triang" is the function which generates the signal. I immediately scaled the results by dividing by the number of samples (100) and multiplying each term except the 0th by 2. Next were the frequencies freq1 = fft.fftfreq(space.shape[-1], d=1/fs). The resulting coefficients (the real parts pertaining to an) are: a0 = 0.5051 a1 = 0.4091 a3 = 0.0452 a5 = 0.0161  As you can see, the absolute values are correct, but the signs are not. What am I missing?   And one bonus question if I may - when doing it this way (using scipy) and plotting each wave separately, should I add the phase angle from np.angle(fft1[n]) into each term, like a1×cos(n Pi t/T + theta) + b1×sin(n Pi t/T + theta)? I'd say yes, but in this example all the bn are 0, and so are the phase angles, so I couldn't really test it.

Answer: Note that when you call fft(), you don’t pass in the time axis (space). So how is the function supposed to know you sampled from -1 to 1? It doesn’t! It assumes you sampled at 0, 1, 2, 3, … N-1. This means that the FFT function sees the signal as shifted compared to your manual calculation, and a shift means the phase in the frequency domain changes. If you sample your triangle function from 0 to 2, instead of -1 to 1, you should get a matching phase. The difference between sampling with a frequency of 1, or any other frequency (ie samples not being at integer locations) is taken care of by fftfreq(), which just tells you the frequency that corresponds to each bin given your sampling frequency.
Question: When display a DataFrame in Jupyter Notebook, if the string value is too long, the last characters will be omitted: df = pd.DataFrame({'A':['ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ']}) display(df)  output:      A     0 ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRST...     I want to change the behavior and only ommit the middle characters of the string when it's too long:      A     0 ABCDEFGHIJKLMNOPQRSTUVW...DEFGHIJKLMNOPQRSTUVWXYZ     Is it possible?

Answer: From the documentation, you can build a custom formatter. def formatter(x): if isinstance(x, str): if len(x) > pd.options.display.max_colwidth: display_width = pd.options.display.max_colwidth split = int((display_width-3) / 2) if display_width % 2: # if odd number return x[:split] + "..." + x[-split:] else: return x[:(split+1)] + "..." + x[-split:] else: return x return None  And use it as such: df.style.format(formatter)       A     0 ABCDEFGHIJKLMNOPQRSTUVW...DEFGHIJKLMNOPQRSTUVWXYZ     This is only a display format so the data is not changed.
Question: Consider a sample dateframe df = pd.DataFrame({'group' : [1, 2, 2], 'x' : [1, 2, 3], 'y' : [2, 3, np.nan]})  If I want to get the max value of variable 'y' without skipping NANs, I would use the function: df.y.max(skipna = False)  The returned results is nan as expected However, if I want to calculate the grouped max value by 'group', as follows: df.groupby('group').y.max(skipna = False)  I got an error message: TypeError: max() got an unexpected keyword argument 'skipna' Seems like the DataFrameGroupBy.max() does not have the argument to skip nas. What would be the best way to get the desired result?

Answer: You can try to apply pd.Series.max: x = df.groupby("group")["y"].apply(pd.Series.max, skipna=False) print(x)  Prints: group 1 2.0 2 NaN Name: y, dtype: float64
Question: How does one convert a CursorResult-object into a Pandas Dataframe? The following code results in a CursorResult-object: from sqlalchemy.orm import Session from sqlalchemy import create_engine engine = create_engine(f"mssql+pyodbc://{db_server}/{db_name}?trusted_connection=yes&driver={db_driver}") q1 = "SELECT * FROM my_schema.my_table" with Session(engine) as session: results = session.execute(q1) session.commit() type(results) >sqlalchemy.engine.cursor.CursorResult  As I couldn't find a way to extract relevant information from CursorResult, it attempted the following instead: # Extracting data as we go with Session(engine) as session: results = session.execute(q1) description = results.cursor.description rows = results.all() session.commit() # Extracting column names colnames = [elem[0] for elem in description] # Extracting types types = [elem[1] for elem in description] # Creating dataframe import pandas as pd pd.DataFrame(rows, columns=colnames)  But what about the dtypes? It doesn't work if I just put them in, though it looks like they are all python types. For my use case I MUST use Session, so I cannot use the first suggestion of doing the classic: # I cannot use pandas.read_sql(q1, engine)  The reason for this is that I have to do multi-batch queries within the same context, which is why I am using the Session class.

Answer: IIUC, just use pd.DataFrame constructor. dtypes are correctly set. # sqlalchemy==2.0.16 # pandas==2.0.2 from sqlalchemy.sql import text with Session(engine) as session: results = session.execute(text(q1)) df = pd.DataFrame(results) # session.commit() # commit is irrelevant if you don't write data  Test on my database: >>> df.head() Scenario Attribute Process Period Region Vintage PV 0 WithHHP16HinsHE0CCS109LHP VAR_Cap EVTRANS_H-L 2014 FR None 296.071141 1 WithHHP16HinsHE0CCS109LHP VAR_Cap EVTRANS_H-M 2014 FR None 11.770909 2 WithHHP16HinsHE0CCS109LHP VAR_Cap IMPELCHIGA 2014 FR None 11851.674497 3 WithHHP16HinsHE0CCS109LHP VAR_Cap EVTRANS_H-L 2015 FR None 296.071141 4 WithHHP16HinsHE0CCS109LHP VAR_Cap EVTRANS_H-M 2015 FR None 11.770909 >>> df.dtypes Scenario object Attribute object Process object Period int64 Region object Vintage object PV float64 dtype: object  Edit: rec = results.fetchone() >>> rec ('WithHHP16HinsHE0CCS109LHP', 'VAR_Cap', 'EVTRANS_H-L', 2014, 'FR', None, 296.071141357762) # python int --^ python float --^ >>> type(rec) sqlalchemy.engine.row.Row
Question: In my code I am trying to achieve the following:  I get each result as soon as any of the processes finish Next iteration must only be called whenever it is necessary (if it is converted into a list, I will have RAM issues)  To my knowledge imap from multiprocessing module should be perfect for this task, but this code: import os import time def txt_iterator(): for i in range(8): yield i print('iterated', i) def func(x): time.sleep(5) return x if __name__ == '__main__': import multiprocessing pool = multiprocessing.Pool(processes=4) for i in pool.imap( func, txt_iterator() ): print('P2', i) pool.close()  Has this output: iterated 0 iterated 1 ... iterated 7 # 5 second pause P2 0 P2 1 P2 2 P2 3 # 5 second pause P2 4 P2 5 P2 6 P2 7  Meaning that it iterates through the whole iterable and only then starts assigning tasks to processes. As far as I could find in the docs, this behavior is only expected from .map (the iteration part). The expected output is (may vary because they run concurrently, but you get the idea): iterated 0 ... iterated 3 # 5 second pause P2 0 ... P2 3 iterated 4 ... iterated 7 # 5 second pause P2 4 ... P2 7  I am sure that I am missing something here but in case I completely misunderstand how this function works, I would appreciate any alternative that will work as intended.

Answer: imap doesn't guarantee consuming the input iterator at the same pace the workers finish their tasks. You can use a threading.BoundedSemaphore (even if you're only using a single thread) to have the input generator wait until the for loop has consumed an item: import multiprocessing import threading import time def txt_iterator(sem: threading.BoundedSemaphore): for i in range(30): sem.acquire() yield i print("iterated", i) def func(x): print("starting work on", x) time.sleep(1) return x if __name__ == "__main__": sem = threading.BoundedSemaphore(4) pool = multiprocessing.Pool(processes=4) for i in pool.imap(func, txt_iterator(sem)): sem.release() print("P2", i) pool.close()
Question: I'm about to create an API using FastApi wherein I have to search for 'user_name' in db. If 'user_name' exists then I have to update the user_details. If the 'user_name' doesn't exist, then I have to create entry for the user. In this case, I think both PUT and POST methods need to be applied for the same API endpoint. Is this possible? can anyone brief me how to do it?

Answer: Yes, it is possible. You just need to use different decorator for the same path. from fastapi import FastAPI app = FastAPI() @app.put("/") async def put_root(): return {"message": "Hello World from put"} @app.post("/") async def post_root(): return {"message": "Hello World from post"}  the code is oversimplify but it shows the concept.
Question: Is it possible to copy a dataframe in the middle of a method chain to a new variable? Something like: import pandas as pd df = (pd.DataFrame([[2, 4, 6], [8, 10, 12], [14, 16, 18], ]) .assign(something_else=100) .div(2) .copy_to_new_variable(df_imag) # Imaginated method to copy df to df_imag. .div(10) )  print(df_imag) would then return:  0 1 2 something_else 0 1.0 2.0 3.0 50.0 1 4.0 5.0 6.0 50.0 2 7.0 8.0 9.0 50.0  .copy_to_new_variable(df_imag) could be replaced by df_imag = df.copy() but this would result in compromising the method chain.

Answer: Actually, this is what I was looking for. Check the link, the idea is from Matt Harrison (who wrote multiple books about pandas) for debugging of method chains. This way is also recommended in this great article 4 Pandas Anti-Patterns to Avoid and How to Fix Them by Aidan Cooper. import pandas as pd def to_df(df, name): globals()[name] = df.copy() return df df = (pd.DataFrame([[1, 2, 3], [10, 10, 10], ], columns=["A", "B", "C"] ) .set_index("C") .pipe(to_df, "df_imag") .sum() )  df_imag is then the intermediate dataframe as described in the question. In jupyter notebooks, if you would like to view the dataframe midway through the chain without interrupting the rest of the chain, you can use .pipe(lambda df_: display(df_) or df_), also explained in the mentioned article: import pandas as pd df = ( pd.DataFrame( [ [2, 4, 6], [8, 10, 12], [14, 16, 18], ] ) .assign(something_else=100) .div(2) .pipe(lambda df_: display(df_) or df_) .div(10) )
Question: I like to have my package installable with pip install ... and to use the pyproject.toml standard. I can specify dependencies to install from git, with: dependencies = [ 'numpy>=1.21', 'psychopy @ git+https://github.com/psychopy/psychopy', ]  But how can I specify a dependency to install from a different indexer, equivalent to: python -m pip install --pre --only-binary :all: -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy scipy With or without the pre-release flag? And how can I specify a dependency to install from a URL, e.g. https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-22.04/wxPython-4.2.1-cp310-cp310-linux_x86_64.whl I tried with no luck: dependencies = [ 'wxPython @ https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-22.04/wxPython-4.2.1-cp310-cp310-linux_x86_64.whl; python_version == "3.10"; sys_platform == "linux"' ] 

Answer: I recommend reading the article "install_requires vs. requirements files" in the "Python packaging user guide". This article is partly outdated but the principle of abstract dependencies vs. concrete dependencies is unchanged. Concrete dependencies do not belong in packaging metadata, i.e. concrete dependencies can not be added to the dependencies list in the [project] table of the pyproject.toml file. As you have noted, it is possible to add "direct references" (for example psychopy @ git+https://github.com/psychopy/psychopy), but not all packaging tools support this and notably PyPI rejects the upload of packages containing such dependencies. What I often recommend, is that if there are dependencies that are not on PyPI then the documentation should reflect that very clearly and prominently. Where to get the dependencies and some examples of how to install them should be in the documentation, for example the URLs of the alternative package repositories. And another thing that I recommend on top of the documentation is to provide one or more requirements.txt files that contain a list of concrete dependencies that have been tested and are known to work well. For example: MyPackage psychopy @ git+https://github.com/psychopy/psychopy wxPython @ https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-22.04/wxPython-4.2.1-cp310-cp310-linux_x86_64.whl; python_version == "3.10"; sys_platform == "linux"  And if I am not mistaken pip can install directly from the URL of such a requirements file: python -m pip install --requirement https://host.example/my-package/requirements.txt so such a command could be added to the documentation as well. Related:  https://stackoverflow.com/a/76548420