eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: While finetuning a model using HF's trainer. training_args = TrainingArguments(output_dir=data_dir + "test_trainer") metric = load_metric("accuracy") def compute_metrics(eval_pred): logits, labels = eval_pred predictions = np.argmax(logits, axis=-1) return metric.compute(predictions=predictions, references=labels) training_args = TrainingArguments(num_train_epochs=5,per_device_train_batch_size=64,per_device_eval_batch_size=32,output_dir="test_trainer", evaluation_strategy="epoch") trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=val_dataset, compute_metrics=compute_metrics, ) trainer.train()  How would I be able to plot the loss in a notebook? (Perhaps Is it possible to get a list of the loss)

Answer: It is possible to get a list of losses. You can access the history of logs after training is complete with: trainer.state.log_history
Question: I am working on time series analysis and I have sales data (lets call it df_panel as we panel data structure) for 700 individual areas for each month of 2021. e.g.     Area Month Sales     Area 1 January 1000   Area 1 February 2000   Area 1 Marts 3000   Area 2 January 1000   Area 2 February 2000   Area 2 Marts 1400   Area 3 January 1000   Area 3 February 1200   Area 3 Marts 1400     Normally when working on sales data you use e.g. ADF Testing to check for unit roots in the sales data. I know how to do this in Python for a standard non-panel data structure using e.g. the adfuller function from statsmodels on a dataframe df: adf_test_result = adfuller(df["Sales"])[1]  How can I do something similar for my panel data structure, as it consists of 700 individual sales curves (one for each area). The goal is to use Panel Data Regression (Fixed or Random Effects) One approximation could be to sum up my panel data sales curve to one sales curve and do the ADF test on that: adf_test_result = adfuller(df_panel.groupby("Month").sum()["Sales"])  But I think this will greatly overestimate the probability of a unit root in the sales data. A lot of information in the sales data is lost when summing it up like this for 700 individual areas. Another approximation could maybe be to check for unit roots in each individual area and somehow take the mean (?) Not exactly sure what is best here... In R there is package plm with function purtest that implements several testing procedures that have been proposed to test unit root hypotheses with panel data, e.g., "levinlin" for Levin, Lin and Chu (2002), "ips" for Im, Pesaran and Shin (2003), "madwu" for Maddala and Wu (1999), and "hadri" for Hadri (2000). Does anyone know how to estimate the unit root for panel data structures? And how to implement this in Python?

Answer: The SAS documentation website HERE tells us that the IPS method uses the average of the ADF test statistics across groups/panels. The ADF test is available from the package "statsmodel" library HERE, so you can simply calculate the tau-statistics yourself, take the average, and calculate the p-value using a t-test. # p-value for a 2-sided t-test from scipy import stats 2*(stats.t.sf( abs(tau_avg) ,dof=1000 ))  Note that 1000 is just an example for a high degree of freedom.
Question: I am upgrading a library to a recent version of SQLAlchemy and I am getting this error  type object 'Base' has no attribute '_decl_class_registry'  On line Base = declarative_base(metadata=metadata) Base._decl_class_registry  How can I solve this?

Answer: Had the same problem. Because of my upgrade of sqlalchemy looks like there is a change in the base code. use this instead to accomplish the same Base.registry._class_registry.values()
Question: I work on several python packages that contain data within them. I add them via the MANIFEST.in file, passing include_package_data=True to setup. For example: # MANIFEST.in graft mypackage/plugins graft mypackage/data  Up to now, this has worked without warnings as far as I know. However, in setuptools 62.3.0, I get the following message: SetuptoolsDeprecationWarning: Installing 'mypackage.plugins' as data is deprecated, please list it in `packages`. 07:53:53 !! 07:53:53 07:53:53 07:53:53 ############################ 07:53:53 # Package would be ignored # 07:53:53 ############################ 07:53:53 Python recognizes 'mypackage.plugins' as an importable package, however it is 07:53:53 included in the distribution as "data". 07:53:53 This behavior is likely to change in future versions of setuptools (and 07:53:53 therefore is considered deprecated). 07:53:53 07:53:53 Please make sure that 'mypackage.plugins' is included as a package by using 07:53:53 setuptools' `packages` configuration field or the proper discovery methods 07:53:53 (for example by using `find_namespace_packages(...)`/`find_namespace:` 07:53:53 instead of `find_packages(...)`/`find:`). 07:53:53 07:53:53 You can read more about "package discovery" and "data files" on setuptools 07:53:53 documentation page.  I get the above warning for pretty much every directory within mypackage that contains data and is included by MANIFEST.in. My goal is to include arbitrary data (which could even include python files in the case of a plugin interface) in a package so that it can be accessed by users who install via wheel or tarball. I would also like that applications built by, e.g., pyinstaller, that pull my package in can easily collect the data with collect_data_files, which for me has worked without any additional setup with the current methodology. What is the proper way to do this going forward?

Answer: The TL;DR is that in Python since PEP 420, directories count as packages, even if they don't have a __init__.py file. The main difference is that directories without __init__.py are called "namespace packages". Accordingly, if a project wants to distribute directories without a __init__.py file, it should use packages=find_namespace_packages() (setup.py) or packages = find_namespace: (setup.cfg). Details on how to use those tools can be found on these docs. Doing this change should make the error go away. The MANIFEST.in or the include_package_data=True should be fine.
Question: Here is the code which can reproduce the problem (it is just for reproducing the problem, so what it does is a bit meaningless): from joblib import Parallel, delayed import tkinter as tk def f(): print('func call') if __name__ == '__main__': root = tk.Tk() button = tk.Button(root, command=lambda: Parallel(n_jobs=-1, backend='threading')(delayed(f)() for _ in range(1)), text='func') button.pack() root.mainloop()  The above code can run perfectly in my IDE, but once I create an executable with Pyinstaller, it bugs. The command line to create the executable is as below: pyinstaller -F main.py  The excepted behavior is that, every time when presses the button in the tkinter window, a func call string should be printed in the terminal. But when I use the executable file to run, every time I press the button, besides the printing in the terminal, a new tkinter window is created. I have also tried to build the executable in Windows with the same command. The executable file runs fine in Windows (No new tkinter window is created when pressing the button). Only the MacOS platform has this problem. How should I fix this? Here is the platform that I have the problem:  CPU: ARM64 (M1) Python: 3.10 joblib: 1.1.0 pyinstaller 5.0.1 

Answer: The problem is solved by adding multiprocessing.freeze_support() to the code. The fixed version of code is as below: import multiprocessing multiprocessing.freeze_support() from joblib import Parallel, delayed import tkinter as tk def f(): print('func called') if __name__ == '__main__': root = tk.Tk() button = tk.Button(root, command=lambda: Parallel(n_jobs=-1, backend='threading')(delayed(f)() for _ in range(1)), text='func') button.pack() root.mainloop()  Thank you rokm for answering my question on GitHub, here is the URL to rokm's answer: https://github.com/pyinstaller/pyinstaller/issues/6852#issuecomment-1138269358.
Question: Consider this simple Python script: $ cat test_utc.py from datetime import datetime for i in range(10_000_000): first = datetime.utcnow() second = datetime.utcnow() assert first <= second, f"{first=} {second=} {i=}"  When I run it from the shell like python test_utc.py it finishes w/o errors, just as expected. However, when I run it in a Docker container the assertion fails: $ docker run -it --rm -v "$PWD":/code -w /code python:3.10.4 python test_utc.py Traceback (most recent call last): File "/code/test_utc.py", line 7, in <module> assert first <= second, f"{first=} {second=} {i=}" AssertionError: first=datetime.datetime(2022, 5, 24, 19, 5, 1, 861308) second=datetime.datetime(2022, 5, 24, 19, 5, 1, 818270) i=1818860  How is it possible? P.S. a colleague has reported that increasing the range parameter to 100_000_000 makes it fail in the shell on their mac as well (but not for me).

Answer: utcnow refers to now refers to today refers to fromtimestamp refers to time, which says:  While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.  The utcnow code also shows its usage of time: def utcnow(cls): "Construct a UTC datetime from time.time()." t = _time.time() return cls.utcfromtimestamp(t)  Such system clock updates are also why monotonic exists, which says:  Return the value (in fractional seconds) of a monotonic clock, i.e. a clock that cannot go backwards. The clock is not affected by system clock updates.  And utcnow has no such guarantee. Your computer doesn't have a perfect clock, every now and then it synchronizes via the internet with more accurate clocks, possibly adjusting it backwards. See for example answers here. And looks like Docker makes it worse, see for example Addressing Time Drift in Docker Desktop for Mac from the Docker blog. Excerpt:  macOS doesn’t have native container support. The helper VM has its own internal clock, separate from the host’s clock. When the two clocks drift apart then suddenly commands which rely on the time, or on file timestamps, may start to behave differently  Lastly, you can increase your chance to catch a backwards update when one occurs. If one occurs not between getting first and second but between second and the next first, you'll miss it! Below code fixes that issue and is also micro-optimized (including removing the utcnow middle man) so it checks faster / more frequently: import time from itertools import repeat def function(): n = 10_000_000 reps = repeat(1, n) now = time.time first = now() for _ in reps: second = now() assert first <= second, f"{first=} {second=} i={n - sum(reps)}" first = second function()
Question: Consider this matrix: [0.9, 0.45, 0.4, 0.35], [0.4, 0.8, 0.3, 0.25], [0.5, 0.45, 0.9, 0.35], [0.2, 0.18, 0.8, 0.1], [0.6, 0.45, 0.4, 0.9]  and this list: [0,1,2,3,3]  I want to create a list that looks like the following: [0.9, 0.8, 0.9, 0.1, 0.9]  To clarify, for each row, I want the element of the matrix whose column index is contained in the first array. How can I accomplish this?

Answer: Zip the two lists together as below a=[[0.9, 0.45, 0.4, 0.35],[0.4, 0.8, 0.3, 0.25],[0.5, 0.45, 0.9, 0.35],[0.2, 0.18, 0.8, 0.1],[0.6, 0.45, 0.4, 0.9]] b=[0,1,2,3,3] [i[j] for i,j in zip(a,b)]  Result [0.9, 0.8, 0.9, 0.1, 0.9]  This basically pairs up each sublist in the matrix with the element of your second list in order with zip(a,b) Then for each pair you choose the bth element of a
Question: I'm trying to understand the basic pytorch autograd system: x = torch.tensor(10., requires_grad=True) print('tensor:',x) x.backward() print('gradient:',x.grad)  output: tensor: tensor(10., requires_grad=True) gradient: tensor(1.)  since x is a scalar constant and no function is applied to it, I expected 0. as the gradient output. Why is the gradient 1. instead?

Answer: Whenever you are using value.backward(), you compute the derivative value (in your case value == x) with respect to all your parameters (in your case that is just x). Roughly speaking, this means all tensors that are somehow involved in your computation that have requires_grad=True. So this means x.grad = dx / dx = 1  To add to that: With the automatic differentiation you always ever compute with "constant" values: All your function or networks are always evaluated at a concrete point. And the gradient you get is the gradient evaluated at that same point. There is no symbolic computation taking place. All the information needed for the computation of the gradient is encoded in the computation graph.
Question: Method chaining is a known way to improve code readability and often referred to as a Fluent API [1, 2]. Pandas does support this approach as multiple method calls can be chained like: #!/usr/bin/env python3 # -*- coding: utf-8 -*- import numpy as np import pandas as pd d = {'col1': [1, 2, 3, 4], 'col2': [5, np.nan, 7, 8], 'col3': [9, 10, 11, np.nan], 'col4': [np.nan, np.nan, np.nan, np.nan]} df = ( pd .DataFrame(d) .set_index('col1') .drop(labels='col3', axis=1) ) print(df)  How could I use method chaining if I need to access attributes of the DataFrame returned from the previous function call? To be specific, I need to call .dropna() on a column subset. As the DataFrame is generated from pd.concat() the exact column names are not known a priori. Therefore, I am currently using a two-step approach like this: #!/usr/bin/env python3 # -*- coding: utf-8 -*- import numpy as np import pandas as pd d_1 = {'col1': [1, 2, 3, 4], 'col2': [5, np.nan, 7, 8], 'col3': [9, 10, 11, np.nan], 'col4': [np.nan, np.nan, np.nan, np.nan]} d_2 = {'col10': [10, 20, 30, 40], 'col20': [50, np.nan, 70, 80], 'col30': [90, 100, 110, np.nan]} df_1 = pd.DataFrame(d_1) df_2 = pd.DataFrame(d_2) df = pd.concat([df_1, df_2], axis=1) print(df) dropped = df.dropna(how='any', subset=[c for c in df.columns if c != 'col4']) print(dropped)  Is there a more elegant way based on method chaining? .dropna() can certainly be chained, but I did not find a way to access the column names of the DataFrame resulting from the previous pd.concat(). I image something like # pseudo-code dropped = ( pd .concat([df_1, df_2], axis=1) .dropna(how='any', subset=<access columns of dataframe returned from previous concat and ignore desired column>) ) print(dropped)  but did not find a solution. Memory-efficiency could be improved by using .dropna() with the inplace=True option to re-assign the variable in-place. However, readability with respect to method chaining remains unimproved.

Answer: Use pipe: dropped = ( pd .concat([df_1, df_2], axis=1) .pipe(lambda d: d.dropna(how='any', subset=[c for c in d.columns if c != 'col4'])) )  output:  col1 col2 col3 col4 col10 col20 col30 0 1 5.0 9.0 NaN 10 50.0 90.0 2 3 7.0 11.0 NaN 30 70.0 110.0  NB. alternative syntax for the dropna: lambda d: d.dropna(how='any', subset=d.columns.difference(['col4']))
Question: import polars as pl df = pl.DataFrame({ "tags": ["a", "a", "a", "b", "c", "c", "c", "c", "d"] })  This is how to compute the most frequent element of the column using the .mode expression: df.select([ pl.col("tags").mode().alias("mode"), ])  How can I display also the frequency/count of that mode?

Answer: There is a value_counts expression. This expression will return a Struct datatype where the first field is the unique value and the second field is the count of that value. df.select([ pl.col("tags").value_counts() ])  shape: (4, 1) ┌───────────┐ │ tags │ │ --- │ │ struct[2] │ ╞═══════════╡ │ {"c",4} │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ {"a",3} │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ {"b",1} │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ {"d",1} │ └───────────┘  Or if you want to have that result as a DataFrame: (df.select([ pl.col("tags").value_counts() ]).to_series().struct.to_frame())  shape: (4, 2) ┌──────┬────────┐ │ tags ┆ counts │ │ --- ┆ --- │ │ str ┆ u32 │ ╞══════╪════════╡ │ c ┆ 4 │ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ a ┆ 3 │ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ d ┆ 1 │ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ b ┆ 1 │ └──────┴────────┘  Edited: Which can be even simpler: df["tags"].value_counts()
Question: I know how to pass fixed arguments in the launch.json, e.g. In Visual Studio Code, how to pass arguments in launch.json . What I really need is a prompt where I can give a value for an argument that changes. In addition, my argument is a (data) directory for which there is a very ugly long absolute path. I'd really like to be able to set the working directory to a path which contains each of my individual data directories so I only need to provide a relative directory path, i.e. just the directory name. I'm working with Python, on Windows (not my choice) using VS Code 1.55.2 (not my choice, either).

Answer: You can use input variables { "version": "0.2.0", "configurations": [ { "name": "Python: Current File with arguments", "type": "python", "request": "launch", "program": "${file}", "args": [ "--dir", "/some/fixed/dir/${input:enterDir}" ] } ], "inputs": [ { "id": "enterDir", "type": "promptString", "description": "Subdirectory to process", "default": "data-0034" } ] }  You can place the ${input:enterDir} in any string in the task "configurations" like the "cwd" property. If you like to pick a directory from a list because it is dynamic you can use the extension Command Variable that has the command pickFile Command Variable v1.36.0 supports the fixed folder specification. { "version": "0.2.0", "configurations": [ { "name": "Python: Current File with arguments", "type": "python", "request": "launch", "program": "${file}", "args": [ "--dir", "${input:pickDir}" ] } ], "inputs": [ { "id": "pickDir", "type": "command", "command": "extension.commandvariable.file.pickFile", "args": { "include": "**/*", "display": "fileName", "description": "Subdirectory to process", "showDirs": true, "fromFolder": { "fixed": "/some/fixed/dir" } } } ] }  On Unix like systems you can include the folder in the include glob pattern. On Windows you have to use the fromFolder to convert the directory path to a usable glob pattern. If you have multiple folders you can use the predefined property.
Question: This is how to source a local image file to the <img> element in html: <html> <h1>This is an image</h1> <img src="file:///C:/Users/MyUser/Desktop/Plotly_Dash_logo.png" alt="image"></img> </html>  This displays the image as expected. But when I try to make the same page using the plotly dash wrapper elements, it does not work: import dash from dash import html, dcc app = dash.Dash(__name__) app.layout = html.Div([ html.H1('This is an image'), html.Img(src=r'file:///C:/Users/MyUser/Desktop/Plotly_Dash_logo.png', alt='image'), ]) if __name__ == '__main__': app.run_server(host='0.0.0.0', port=8080, debug=False, use_reloader=False)  The local image file does not display. But if I replace the source with a file from the internet, like 'https://rapids.ai/assets/images/Plotly_Dash_logo.png', it works just fine. What is going on here?

Answer: After some more searching, I found that I could place my image file in a folder named "assets/", then reference it relative to the app folder. html.Img(src=r'assets/Plotly_Dash_logo.png', alt='image')  I could also use a special method of the app instance dash.Dash.get_asset_url(). html.Img(src=app.get_asset_url('my-image.png'))  Source: https://dash.plotly.com/dash-enterprise/static-assets
Question: I am trying to use BERTopic to analyze the topic distribution of documents, after BERTopic is performed, I would like to calculate the probabilities under respective topics per document, how should I did it? # define model model = BERTopic(verbose=True, vectorizer_model=vectorizer_model, embedding_model='paraphrase-MiniLM-L3-v2', min_topic_size= 50, nr_topics=10) # train model headline_topics, _ = model.fit_transform(df1.review_processed3) # examine one of the topic a_topic = freq.iloc[0]["Topic"] # Select the 1st topic model.get_topic(a_topic) # Show the words and their c-TF-IDF scores  Below is the words and their c-TF-IDF scores for one of the Topics image 1 How should I change the result into Topic Distribution as below in order to calculate the topic distribution score and also identify the main topic? image 2

Answer: First, to compute probabilities, you have to add to your model definition calculate_probabilities=True (this could slow down the extraction of topics if you have many documents, > 100000). # define model model = BERTopic(verbose=True, vectorizer_model=vectorizer_model, embedding_model='paraphrase-MiniLM-L3-v2', min_topic_size= 50, nr_topics=10, calculate_probabilities=True)  Then, calling fit_transform, you should save the probabilities: headline_topics, probs = model.fit_transform(df1.review_processed3)  Now, you can create a pandas dataframe which shows probabilities under respective topics per document. import pandas as pd probs_df=pd.DataFrame(probs) probs_df['main percentage'] = pd.DataFrame({'max': probs_df.max(axis=1)})
Question: I installed easyocr in a newly created python environment using pip install easyocr. Then i installed opencv-python. when i try to execute the code - import cv2 img = cv2.imread('2.jpg') cv2.imshow('sd',img) cv2.waitKey(0)  It's giving error  OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\highgui\src\window.cpp:1268: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvShowImage' 

Answer: Problem: If you already have an existing OpenCV version in your system/environment; installing easyOCR can alter that. Going through the requirements.txt file of easyOCR, opencv-python-headless gets installed. The following excerpt is taken from opencv-python-headless documentation:  Packages for server (headless) environments (such as Docker, cloud environments etc.), no GUI library dependencies   These packages are smaller than the two other packages above because they do not contain any GUI functionality (not compiled with Qt / other GUI components). This means that the packages avoid a heavy dependency chain to X11 libraries and you will have for example smaller Docker images as a result. You should always use these packages if you do not use cv2.imshow et al. or you are using some other package (such as PyQt) than OpenCV to create your GUI.  TLDR; In short, easyocr disables existing GUI capabilities. It has been designed exclusively for containerized applications and/or server deployments. Solution: To use easyOCR with OpenCV, you can try either one of the following: 1. Change installation sequence: All the following can be done using pip:  First install easyocr. Uninstall opencv-python-headless. Install opencv-python  2. Use matplotlib One can still display images using matplotlib: import cv2 from matplotlib import pyplot as plt img = cv2.imread('img.jpg',0) plt.imshow(img) plt.show()
Question: I have a Django project, and I want to save created_at datetime in the database. I generate datetime.now with jdatetime (or Khayyam) python package and try to save this in DateTimeField. But sometimes it raises error because the Gregorian(miladi) date of the entry does not exist. what can I do about this?

Answer: In my idea, you can save two model fields. One is DateTimeField contains gregorian datetime, and another one, CharField contains converted Jalali to a String value and save it. The DateTimeField for functionality, e.g., filter between to datetime. The StringField for representing in response(without overload).