eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here's a Python program which does the following:  Makes an API call to treasury.gov to retrieve data Stores the data in a Pandas dataframe Plots the data as a bar chart  import requests import pandas as pd import matplotlib.pyplot as plt date = '1900-01-01' transaction_type = 'Withdrawals' transaction_catg = 'Interest on Treasury Securities' page_size = 10000 url = 'https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/dts/deposits_withdrawals_operating_cash' url_params = f'?filter=record_date:gt:{date},transaction_type:eq:{transaction_type},transaction_catg:eq:{transaction_catg}&page[size]={page_size}' response = requests.get(url + url_params) result_json = response.json() df = pd.DataFrame(result_json['data']) # Convert transaction_today_amt to numeric df['transaction_today_amt'] = pd.to_numeric(df['transaction_today_amt']) df.plot.bar(x='record_date', y='transaction_today_amt') plt.show()  Here's the resulting chart that I get:  As you can see, there are too many x-axis labels. Question What's a good way to setup the chart so that the x-axis labels are legible?

Answer:  The 'record_date' values are strings, not datetimes.  df.record_date = pd.to_datetime(df.record_date)   Typically, a line plot should be used for continuous timeseries data.  Line Plot ax = df.plot(x='record_date', y='transaction_today_amt', figsize=(12, 7))   Scatter Plot ax = df.plot(kind='scatter', x='record_date', y='transaction_today_amt', marker='.', figsize=(12, 7))   Bar Plot  Use matplotlib directly, because the xticks are datetime ordinals, which will label the x-axis with the Year.  ax.get_xticks() → array([12418., 13149., 13879., 14610., 15340., 16071., 16801., 17532., 18262., 18993., 19723.])    fig, ax = plt.subplots(figsize=(8, 5), dpi=200) ax.bar(x='record_date', height='transaction_today_amt', data=df) ax.set_yscale('log')    pandas treats the xticks as discrete, and plots all of them.  ax = df.plot(kind='bar', x='record_date', y='transaction_today_amt', figsize=(12, 7))
Question: I can't seem to find this anywhere else, and I could be wording the question incorrectly, but I am getting the value counts for the 'Type' column in adf grouped by account ID and getting results like so: print(df_activities.groupby(['Account ID'])['Type'].value_counts().sort_index()) Account ID Type 0011N000017WOso Call 5 Email 119 Null 2 Outreach 65 Unclassified 1 ... 001o0000017a8o5 Email 46 LinkedIn 11 Null 2 Others 6 Outreach 8  What I would like is to only extract Call, Email, LinkedIn, and Meeting from the value counts, but if an ID is missing one of these values entirely, I need it to show that there are 0 present. For example, this would be the ideal output of the above: Account ID Type 0011N000017WOso Call 5 Email 119 LinkedIn 0 Meeting 0 ... 001o0000017a8o5 Email 46 Call 0 LinkedIn 11 Meeting 0  I want to map these to separate columns afterwards, with something like this: df_acc['Email Activity'] = df_acc['Account ID'].map(df_contacts.groupby(['Account ID'])['Type']='Email'.sum()) 

Answer: Instead of value_counts, use crosstab and stack: out = pd.crosstab(df['Account ID'], df['Type']).stack()  For only specific types: types = ['Call', 'Email', 'LinkedIn', 'Meeting'] out = (pd.crosstab(df['Account ID'], df['Type']) .reindex(columns=types, fill_value=0).stack() )  Output: Account ID Type 0011N000017WOso Call 5 Email 119 LinkedIn 0 Meeting 0 001o0000017a8o5 Call 0 Email 46 LinkedIn 11 Meeting 0 dtype: int64  Variant that might be a bit more efficient if there are many non-target categories: types = ['Call', 'Email', 'LinkedIn', 'Meeting'] m = df['Type'].isin(types) out = (pd.crosstab(df.loc[m, 'Account ID'], df.loc[m, 'Type']) .reindex(columns=types, fill_value=0).stack() )
Question: I am drawing a line chart using matplotlib as shown below. import matplotlib.pyplot as plt import pandas as pd import io temp = u""" tenor,yield 1M,5.381 3M,5.451 6M,5.505 1Y,5.393 5Y,4.255 10Y,4.109 """ data = pd.read_csv(io.StringIO(temp), sep=",") plt.plot(data['tenor'], data['yield'])  Output: The tick intervals on the x-axis are all the same.  What I want : Set the tick interval of the x-axis differently as shown in the screen below Is there any way to set tick intvel differently? 

Answer: In the column 'tenor', 'M' represents month and 'Y' represents year. Create a 'Month' column with 'Y' scaled by 12 Months. It's more concise to plot the data directly with pandas.DataFrame.plot, and use .set_xticks to change the xtick-labels. Tested in python 3.12.0, pandas 2.1.1, matplotlib 3.8.0 data = pd.read_csv(io.StringIO(temp), sep=",") # Add a column "Month" from the the column "tenor" data["Month"] = data['tenor'].apply(lambda x : int(x[:-1]) *12 if 'Y' in x else int(x[:-1])) # plot yield vs Month ax = data.plot(x='Month', y='yield', figsize=(17, 5), legend=False) # set the xticklabels _ = ax.set_xticks(data.Month, data.tenor)  The output:   .apply with a lambda function is fastest  Given 6M rows  data = pd.DataFrame({'tenor': ['1M', '3M', '6M', '1Y', '5Y', '10Y'] * 1000000})  Compare Implementations with %timeit in JupyterLab  .apply & lambda  %timeit data['tenor'].apply(lambda x: int(x[:-1]) *12 if 'Y' in x else int(x[:-1])) 2 s ± 50.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   .apply with a function call  def scale(x): v = int(x[:-1]) return v * 12 if 'Y' in x else v %timeit data['tenor'].apply(scale) 2.02 s ± 20.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   Vectorized np.where with assignment expression  %timeit np.where(data.tenor.str.contains('Y'), (v := data.tenor.str[:-1].astype(int)) * 12 , v) 2.44 s ± 26.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   Vectorized np.where without assignment expression  %timeit np.where(data.tenor.str.contains('Y'), data.tenor.str[:-1].astype(int) * 12 , data.tenor.str[:-1].astype(int)) 3.36 s ± 5.38 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I just installed the latest version (1.83.1) of VSC on Ubuntu. I plan to develop in Python, and am just getting started using Linters. I have installed the VSC extensions for Python, including the Pylance extension. I also installed the pylint and flake8 extensions as I wanted to compare the two linters. All documentation I've seen, including that on Microsoft's web site, is that I should be able to select which linter to use by entering the command (ctrl-shft-p) "python: select linter". However that command does not does not appear in the command list. Both linter extensions are enabled, and both appear to work, as I get pop up messages with lint corrections. Has there been a change in the linter selection that does not appear in the documentation?

Answer: The Python: Select Linter command was removed in the 2023.18.0 release / between that release and the 2023.16.0 release. You can see the command definition and the command palette menu item definition still there in the 2023.16.0 release's extension manifest (search "setLinter"), whereas it's gone in the 2023.18.0 extension manifest. For more info on why this change (probably) happened, see Why does the VS Code Python extension (circa v2018.19) no longer provide support for several Python tools like linters and formatters?. TL;DR the Python extension is deferring linting to separate extensions now.
Question: I am trying to group by a polars dataframe following the document: https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/api/polars.DataFrame.group_by.html#polars.DataFrame.group_by import polars as pl df = pl.DataFrame( { "a": ["a", "b", "a", "b", "c"], "b": [1, 2, 1, 3, 3], "c": [5, 4, 3, 2, 1], } ) df.group_by("a").agg(pl.col("b").sum())  However, I am getting this error: AttributeError: 'DataFrame' object has no attribute 'group_by' 

Answer: On checking I found my polars version : pl.__version__ 0.17.3  https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/api/polars.DataFrame.groupby.html I need to do: df.groupby("a").agg(pl.col("b").sum()) # there is no underscore in groupby #output shape: (3, 2) a b str i64 "a" 2 "c" 3 "b" 5  and the document says :  Deprecated since version 0.19.0: This method has been renamed to DataFrame.group_by().  This is the new document for polars version 0.19 https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/api/polars.DataFrame.group_by.html#polars-dataframe-group-by
Question: Giving two dataframes df1: col1 ----- 1 1 1 2 3 1 1 2  and df2: colA | colB ------------ 1 | 2 1 | 4 2 | 1 3 | 1 5 | 1 4 | 5  I want to return True if the count of the occurence of every element e in col1 in df1 is equal to the count of the occurence of e in both colA and colB in df2. In the example above for df1 and df2 it should return True. Explanation: count of occurence of 1 in col1 in df1= 5 = count of occurence of 1 in colA and colB in df2. count of occurence of 2 in col1 = 2 = count of occurence of 2 in colA and colB in df2. count of occurence of 3 in col1 = 1 = count of occurence of 3 in colA and colB in df2 Logic: The idea is to groupby the elements of col1 in df1 and count the occurences of each one of it then go search the count of their occurence in colA and colB in df2. I've tried the following code: def records_check(df1, df2): if df1.groupby(['col1']).size() == df2.groupby(['colA', 'colA']).size(): return True else: return False  But I've got this: ValueError: Can only compare identically-labeled Series objects  What is the most efficient way to achieve this especially when dealing with huge data please?

Answer: You can use value_counts on col1 and the stacked version of df2, then compare the outputs: c1 = df1['col1'].value_counts(sort=False) c2 = df2.stack().value_counts(sort=False) # if more columns # c2 = df2[['colA', 'colB']].stack().value_counts() out = c1.eq(c2.reindex_like(c1)).all()  numpy variant: # get values and counts for each dataset n1, c1 = np.unique(df1['col1'], return_counts=True) n2, c2 = np.unique(df2[['colA', 'colB']].to_numpy().ravel(), return_counts=True) # compare the counts for existing values in `df1['col1']` np.array_equal(c1, c2[np.isin(n2, n1)])  Output: True
Question: I am trying to find out how I can use np.where in a dynamic way, where I select some predefined values, pass them to a function and let them create the condition. Ideally I want to create long conditions with several logical operators. In the code below I am stupidly trying to use a string: cond_arr[1]['cond'] as a logical operator, to illustrate what I want, because I can't find out how to proceed with this? Is there an elegant(or just working) way to create these dynamic conditions? import numpy as np import pandas as pd import random from datetime import datetime, timedelta data = { 'open': [random.uniform(50, 100) for _ in range(30)], 'high': [random.uniform(100, 150) for _ in range(30)], 'low': [random.uniform(25, 50) for _ in range(30)], 'close': [random.uniform(50, 100) for _ in range(30)], 'volume': [random.randint(1000, 10000) for _ in range(30)], 'datetime': [datetime(2023, 10, 1, 0, 0) + timedelta(hours=i) for i in range(30)] } df = pd.DataFrame(data) # The meat and potatoes indicators = [{"ind": "open"},{"ind": "rsi"}, {"ind": "macd"}] conds = [{"cond": "<"}, {"cond": ">"}, {"cond": "=="}, {"cond": "and"}, {"cond": "or"}] values = [{"val": 10}, {"val": 100}, {"val": 15}, {"val": 17}, {"val": 18}, {"val": 7}] def create_condition(cond_array): print(f'{cond_array[0]["ind"]}') # Use double curly braces to escape #df["signal"] = np.where(df["open"] > 10, 1, -1) <-- what i want to do below df["signal"] = np.where(df[f'{cond_array[0]["ind"]}'] cond_arr[1]['cond'] df[f'{cond_array[2 ["val"]}'], 1, -1) selected_conds = [indicators[0],conds[0],values[0]] create_condition(selected_conds) 

Answer: With pd.eval to evaluate a dynamic expression: def make_clause(df, col_d, cond_d, val_d): col, cond, val = col_d['ind'], cond_d['cond'], val_d['val'] df["signal"] = np.where(pd.eval(f'df.{col} {cond} {val}', target=df), 1, -1) selected_conds = [indicators[0], conds[0], values[0]] make_clause(df, *selected_conds)   In case if a target column name contains spaces, use f'df["{col}"] {cond} {val}' as an expression.
Question: I wanted to load a model using tf.keras.models.load_model('ACM1_9035P.keras') as I usually do but all of a sudden this time it wouldn't load. I got the error message seen in the title. This method has worked many times before but this time it didn't. What's the problem and how can I fix it? I can provide more information about the model and system if needed. I used google colab to train, save and load the model. The model was saved using model.save() The model is saved as a .keras file

Answer: I found the answer: The model was created and saved using TensorFlow version 2.13.0 I was trying to load it using TensorFlow version 2.14.0 Downgrading the version in Google Collab to 2.13.0 allowed me to load the model as I have done before. To see how you can change the TensorFlow version in Collab, check out this page: https://gist.github.com/mervess/5efbec9f62a55a4cd47c900999db7927 Edit: After deploying many models over different tensorflow versions and OSs, I have found it is much less of a headache to use the .h5 file.
Question: I need to insert within two new lists, objects that are not present in list 'PListaServiceID' but are present in list 'SListaServiceID', the same but in reverse. so I started by doing this PListaServiceName = [] PListaServiceID = [] SListaServiceName = [] SListaServiceID = [] CListaServiceID = [] C2ListaServiceID = [] for i in jsonPResponse["result"][0]["data"]: x = i['dimensionMap'] PListaServiceName.append(x['dt.entity.service.name']) PListaServiceID.append(x['dt.entity.service']) for p in jsonSResponse["result"][0]["data"]: z = p['dimensionMap'] SListaServiceName.append(z['dt.entity.service.name']) SListaServiceID.append(z['dt.entity.service']) writer = pd.ExcelWriter("tabella.xlsx", engine="xlsxwriter") #print("debug: ",NomePrimoMese) CreateExcel(PListaServiceName,PListaServiceID, NomePrimoMese, writer) CreateExcel(SListaServiceName,SListaServiceID, NomeSecondoMese, writer) for c in PListaServiceID: for z in SListaServiceID: if c!=z: CListaServiceID.append(c) CreateExcel(CListaServiceID, SListaServiceID, "Confronto", writer) writer.close()  since I would have to repeat the last for loop also to fill the other new list, can there be a better way to do all this stuff?

Answer: I found a difference, it depends on how you want the final data, in my case I wanted a list with all the repetitions within the list. either way works #Elements present in PListaServiceID but not in SListaServiceID #without repetition CListaServiceID = list(set(PListaServiceName) - set(SListaServiceName)) #with repetition ConfrontoListaService = [ x for x in PListaServiceName if x not in SListaServiceName]
Question: I have a 921000 x 3 numpy array (921k 3D points, one point per row) that I am trying to pack into a protobuf message and I am running into performance issues. I have control over the protocol and can change it as needed. I am using Python 3.10 and numpy 1.26.1. I am using protocol buffers because I'm using gRPC. For the very first unoptimized attempt I was using the following message structure: message Point { float x = 1; float y = 2; float z = 3; } message BodyData { int32 id = 1; repeated Point points = 2; }  And packing the points in one at a time (let data be the large numpy array): body = BodyData() for row in data: body.points.append(Point(x=row[0], y=row[1], z=row[2]))  This takes approximately 1.6 seconds, which is way too slow. For the next attempt I ditched the Point structure and decided to transmit the points as a flat array of X/Y/Z triplets: message Points { repeated float xyz = 1; } message BodyData { int32 id = 1; Points points = 2; }  I did some performance tests to determine the fastest way to append a 2D numpy array to a list, and got the following results: # Time: 80.1ms points = [] points.extend(data.flatten()) # Time: 96.8ms points = [] points.extend(data.reshape((data.shape[0] * data.shape[1],))) # Time: 76.5ms - FASTEST points = [] points.extend(data.flatten().tolist())  From this I determined that .extend(data.flatten().tolist()) was the fastest. However, when I applied this to the protobuf message, it slowed way down: # Time: 436.0ms body = BodyData() body.points.xyz.extend(data.flatten().tolist())   So the fastest I've been able to figure out how to pack the numpy array into any protobuf message is 436ms for 921000 points. This is very far short of my performance target, which is ~12ms per copy. I'm not sure if I can get close to that but, is there any way I can do this more quickly?

Answer: If your goal is just to send something over gRPC to another program that you control, then you don't actually have to convert everything into "native" protobuf messages; you can use a protobuf bytes field to store another serialization format, such as numpy tobytes() output, or Arrow. This will be much faster.
Question: I deployed airflow on kubernetes using the official helm chart. I'm using KubernetesExecutor and git-sync. I am using a seperate docker image for my webserver and my workers - each DAG gets its own docker image. I am running into DAG import errors at the airflow home page. E.g. if one of my DAGs is using pandas then I'll get Broken DAG: [/opt/airflow/dags/repo/dags/airflow_demo/ieso.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/dags/project1/dag1.py", line 7, in <module> from pandas import read_parquet ModuleNotFoundError: No module named 'pandas'  I dont have pandas installed on the webserver or scheduler docker images, because if I understand it correctly you shouldn't install the individual dependencies on these. I am getting the same error when running airflow dags list-import-errors on the scheduler pod. I do have pandas installed on the worker pod, but it doesn't get run, because the DAG cannot be discovered through these errors. How do I make airflow discover this DAG without installing pandas to either scheduler or webserver? I know installing it on both will fix this, however I am not interested in doing it this way.

Answer: It's great crafting an answer laid out by the OP's comments 😜 In the comments, @user430953 provided this link to Airflow's documentation, where it states:  One of the important factors impacting DAG loading time, that might be overlooked by Python developers is that top-level imports might take surprisingly a lot of time and they can generate a lot of overhead and this can be easily avoided by converting them to local imports inside Python callables for example.  This makes sense: The scheduler will monitor the Dag Folder and it will try to load instances of class DAG by executing the .py files it finds in the folder as if they were regular "importable" Python modules (erm... well... because they are) and then "fishing" entities (objects? instances?) of type DAG out the module's variables. That happens, more specifically, here and then here (very broadly speaking and just if you're curious) As with any regular Python module, the Python interpreter will "run" the code on import, so any top level import will be attempted. If the module imports a package that the scheduler's Python interpreter doesn't have, an ImportError will be thrown. Since the OP stated...  I do have pandas installed on the worker pod  ... this means the actual work will be doable. We just need to avoid the scheduler from running this import. The fastest way moving the import statement into the function itself, so the pandas module is only imported when the actual work is performed (unsurprisingly, in the worker machines... which do have Pandas installed) So moving from: import pandas as pd with DAG(...) as dag: @task def some_pandas_work(**context): # ...  to... with DAG(...) as dag: @task def some_pandas_work(**context): import pandas as pd # ...  ... should do the trick
Question: Used to have a Dockerfile with conda and flask that worked locally but after installing pytorch had to switch to micromamba, the old CMD no longer works after switching to image mambaorg/micromamba:1-focal-cuda-11.7.1, this is how my Dockerfile looks like right now: FROM mambaorg/micromamba:1-focal-cuda-11.7.1 WORKDIR /app COPY environment.yml . RUN micromamba env create --file environment.yml EXPOSE 5001 COPY app.py . CMD ["/bin/bash", "-c", "source activate env-app && flask run --host=0.0.0.0 --port=5001"]  Now I get error: /bin/bash: activate: No such file or directory but before when I was using conda it was working, I think the CMD command needs to be updated.

Answer: Assuming these 2 files: environment.yml name: testenv channels: - conda-forge dependencies: - python >= 3.9 - flask  app.py from flask import Flask # Create a Flask application app = Flask(__name__) # Define a route for the root URL @app.route("/") def hello_world(): return "Hello, World!" if __name__ == "__main__": # Run the Flask app on the local development server app.run()   Then in your Dockerfile you don't activate the environmnent but use the path to environment, e.g. /opt/conda/envs/testenv/bin/flask (note the testenv in the path and in the environment.yaml): Dockerfile FROM mambaorg/micromamba:1-focal-cuda-11.7.1 WORKDIR /app COPY environment.yml . RUN micromamba env create --file environment.yml EXPOSE 5001 COPY app.py . CMD micromamba run -n testenv flask run --host=0.0.0.0 --port=5001   To build: docker build . -f Dockerfile -t my/image:name  To run: docker run --rm -i -t -p 5001:5001 my/image:name  Then navigating the browser to localhost:5001 you should see Hello World.
Question: I'm looking to replicate the results of the following MATLAB code in SciPy. MATLAB Version: f = @(x, y) besselh(0, 2, x.^2 + y.^2); integral2(f, -0.1, 0.1, -0.1, 0.1)  The MATLAB result is: ans = 0.0400 + 0.1390i  SciPy Version: f = lambda x, y: scipy.special.hankel2(0, x**2 + y**2) scipy.integrate.dblquad(f, -0.1, 0.1, lambda x: -0.1, lambda x: 0.1)  However, when I run the SciPy code, I receive the following warning: IntegrationWarning: The occurrence of roundoff error is detected ...  And the result is different from the MATLAB output: (nan, 2.22e-15)  Can someone explain why I'm getting this warning and how to achieve a result similar to MATLAB's?

Answer: I believe the issue is that the integral goes through the origin, which produces NaN for H0. This is because H0 = J0 - i*Y0, and Y0 asymptotes the y-axis. For this specific case, you can use the H0 definition above to split the integrals into the real and complex parts. When doing this, you can make sure you don't cross 0 for the complex part of the integral. import scipy j0 = lambda x, y: scipy.special.jv(0, x**2 + y**2) y0 = lambda x, y: scipy.special.yv(0, x**2 + y**2) r1 = scipy.integrate.dblquad(j0, -0.1, 0.1, -0.1, 0.1) r21 = scipy.integrate.dblquad(y0, 1e-10, 0.1, -0.1, 0.1) r22 = scipy.integrate.dblquad(y0, -0.1, -1e-10, -0.1, 0.1) res = r1[0] - 1j*(r21[0]+r22[0]) # 0.039999377783047595+0.13896313686573833j