eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Python project which I created according to basic Poetry instructions. The project folder is something like this: my-project +----my_project | +-- my_project.py | +-- File1.py | +-- File2.py | +----pyproject.toml  Example of how I import stuff from one file to another: in my_project.py I have the code from . import File1, File2  If I want to debug this from VS Code, if I try F5 in the my_project.py, I get the error: Exception has occurred: ImportError attempted relative import with no known parent package  However, if I don't express the imports like above, I can't run it using the poetry command. In the pyproject.toml file, I have this: [tool.poetry.scripts] my-project = "my_project.my_project:run"  run is the entry-point method in the my_project.py file. To run the project from command prompt, I go to the project folder (where the package folder is) and I type poetry run my-project Again, up to this point, everything according to the Poetry documentation. How could I debug this project in VS Code? I know I need to create a launch.json file, but I don't know how the configuration should look.

Answer: For Visual Studio Code, you could try this:  add an __init__.py file in the sub-directory my_project in the .vscode directory, add a lauch.json file with the following content:  { "version": "0.1.0", "configurations": [ { "name": "my-project", "type": "python", "request": "launch", "cwd": "${workspaceFolder}", "module": "my_project", "args": [] } ] }  Here, cwd points to your workspace folder, which should be the parent directory of my-project. You should then be able to run successfully the Run and Debug module of Visual Studio Code. As for Poetry, try modifying your pyproject.toml like this (there seems to be a typo, hyphen vs underscore): [tool.poetry.scripts] my-project = "my-project.my_project:run"  And make sure to set the parent directory of my-project as your current working directory when you run poetry run my-project. See this post for additional guidance.
Question: I have the below dataframe. Column_1 Column_2 Name Xxxx Age 28 Gender M Name yyyy Age 26 Gender F  My expected output is Name Age Gender Xxxx 28 M yyyy 26 F  I tried df.T(), but it's writing each name, age and gender to separate columns. How to achieve the above output in python/pandas.

Answer: Try with groupby and pivot: df["idx"] = df.groupby("Column_1").cumcount() >>> df.pivot("idx", "Column_1", "Column_2").reset_index(drop=True).rename_axis(columns=None) Age Gender Name 0 28 M Xxxx 1 26 F Yyyy
Question: In Python, we need an await keyword before each coroutine object to have it called by the event loop. But when we put await, it makes the call blocking. It follows that we end up doing the same thing as we do in the blocking fashion. What is the point of having such a use? https://www.aeracode.org/2018/02/19/python-async-simplified/ https://stackabuse.com/python-async-await-tutorial/

Answer: await makes the call locally blocking, but the "wait" is transmitted through the async function (which is itself awaited), such that when it reaches the reactor the entire task can be moved to a waitlist and an other can be run instead. Furthermore you do not need an await, you could also spawn the coroutine (to a separate task, which you may or may not wait on), or use one of the "future combinators" (asyncio.gather, asyncio.wait, ...) to run it concurrently with others.
Question: I want to fit a curve with the following data, but I get the error: ipython-input>:2: RuntimeWarning: overflow encountered in exp  Does anyone know what is the reason for this problem? I fitted this curve with a different datatype for Matlab and it worked fine. I used the initial condition from my Matlab code. Both curves are the same but the values of the y-axes are much higher in this case. import numpy as np import scipy.optimize #sympy.init_printing() import matplotlib.pyplot as plt from scipy.optimize import curve_fit list_R1_fit = [ 19.53218114920747, 42.52167990454083, 60.95540646861309, 70.10646960395906, 73.99897337091254, 75.36736639556727, 75.69578522881915, 75.62147077733012, 75.42605227692485, 75.21657113589387, 75.04519265636262, 74.94144261816007, 74.92153132015117, 74.99475606015201, 75.15746897265564 ] tau_list = [ 0.052, 0.12, 0.252, 0.464, 0.792, 1.264, 1.928, 2.824, 4, 5.600, 7.795, 10.806, 14.928, 20.599, 28.000 ] array_R1_fit = np.asarray(list_R1_fit) tau_array = np.asarray(tau_list) plt.plot(tau_array, array_R1_fit, 'o') def func_R1_fit( t, a0, a1, a2, a3, a4): R1_fit_Curve = ( a0 * np.exp( a1 * (1 - (28 / t)**(4 / 5)) ) + a2 * (t / ((a3)**a4 + t**a4)) ) return R1_fit_Curve pars, cov = curve_fit( f=func_R1_fit, xdata=tau_array, ydata=array_R1_fit, p0=[ 0.249714296337621, 0.101851223776512, 0.209343265573669, 0.306273529630680, 1.511897539010256 ], bounds=(-np.inf, np.inf), maxfev=100000 )  I have generated more data in the first part of the chart. Now I get another error. <ipython-input-7-f61377d66140>:2: RuntimeWarning: invalid value encountered in double_scalars R1_fit_Curve=a0*np.exp(a1*(1-(28/t)**(4/5)))+a2*(t/((a3)**a4+t**a4))  The new lists are as follows: list_R1_fit=[8.889450414508385, 13.832704635961235, 3.0955553517738656, 6.944672155278666, 19.53218114920747, 23.06912497313617, 32.92595485184, 42.52167990454083, 54.23640835031421, 60.95540646861309, 66.91996368676925, 70.10646960395906, 72.69136093289741, 73.99897337091254, 74.93277916119311, 75.36736639556727, 75.62190347933995, 75.69578522881915, 75.68268608294542, 75.62147077733012, 75.52270979845973, 75.42605227692485, 75.21657113589387, 75.04519265636262, 74.94144261816007, 74.92153132015117, 74.99475606015201, 75.15746897265564] tau_list=[0.03,0.04,0.052/3,0.052/2,0.052,0.12/2,(0.052+0.12)/2,0.12,(0.252+0.12)/2,0.252,(0.464+0.252)/2,0.464,(0.464+0.792)/2,0.792,(1.264+0.792)/2,1.264,(1.264+1.928)/2,1.928,(2.824+1.928)/2,2.824,(2.824+4)/2,4,5.600,7.795,10.806,14.928,20.599,28.000] 

Answer: If you set the dtypes of array_R1_fit and tau_array to np.longdouble ornp.float64 should fix the RuntimeWarning: overflow encountered in exp that is: array_R1_fit = np.asarray(list_R1_fit, dtype=np.longdouble) tau_array =np.asarray(tau_list, dtype=np.longdouble)  Note that if you are on a Windows 64-bit computer np.longdouble will not result in float128 but is defined to be float64. You can try to run it on a linux system.
Question: I use this example code given in plotly website. import plotly.express as px df = px.data.medals_long() fig = px.bar(df, x="medal", y="count", color="nation", pattern_shape="nation", pattern_shape_sequence=[".", "x", "+"]) fig.show()  This gives a plot like below.  How can I add a text box under the legend in the plot to get it like in the picture below?  I see some examples using Text and Annotations but was wondering if there is any other approach to do it.

Answer: From what I could find, adding an annotation is the only way to get the expected output natively in plotly. An implemntation of the annotation will look like: fig.add_annotation(text='South Korea: Asia <br>China: Asia <br>Canada: North America', align='left', showarrow=False, xref='paper', yref='paper', x=1.1, y=0.8, bordercolor='black', borderwidth=1)  Some additional html attributes you can add to the text attribute be found here: https://plotly.com/chart-studio-help/adding-HTML-and-links-to-charts/
Question: I'm newbie in Apache Airflow. There are a lot of examples of basic DAGs in the Internet. Unfortunately, I didn't find any examples of single-task DAG's. Most of DAG's examples contain bitshift operator in the end of the .py script, which defines tasks order. For example: # ...our DAG's code... task1 >> task2 >> task3  But what if my DAG has just a single task at the moment? My question is - do I need to use this single task's name in the end of Python file? Or if we have only 1 task in the scope, Airflow will handle it itself, and the last line of code below is redundant? from datetime import timedelta from airflow.operators.bash import BashOperator from airflow.utils.dates import days_ago default_args = { 'owner': 'airflow', 'depends_on_past': False, 'email': ['airflow@example.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, } with DAG( 'tutorial', default_args=default_args, description='A simple tutorial DAG', schedule_interval=timedelta(days=1), start_date=days_ago(2), tags=['example'], ) as dag: t1 = BashOperator( task_id='print_date', bash_command='date', ) t1 # IS THIS LINE OF CODE NECESSARY? 

Answer: The answer is NO, you don't need to include the last line. You could also avoid the asignment of the variable t1, leaving the DAG like this: with DAG( 'tutorial', default_args=default_args, description='A simple tutorial DAG', schedule_interval=timedelta(days=1), start_date=days_ago(2), tags=['example'], ) as dag: BashOperator( task_id='print_date', bash_command='date', )  The reason to perfom the assignment of an instance of an Operator (such as BashOperator), to a variable (called Task in this scope) is similiar to any other object in OOP. In your example there is no other "operation" perfomed over the t1 variable (you are not reading it or consuming any method from it) so there no is no reason to declare it. When starting with Airflow, I think is very clarifying to use the DebugExecutor to perform quick tests like this and understand how everything is working. If you are using VS Code you can find an example config file, here.
Question: I created four different versions of conda virtual environments (envs) for image processing tasks. Each env includes GDAL and OpenCV, and some subset of related libs and dependencies. I want to cull my list of image processing envs down to the most recently created one, which will have the most complete set of the libs I use. But I don't remember the order I created the envs. Is there a way to see the creation date of individual envs or perhaps a list of creation dates for all of my conda envs? UPDATE Here is the output of the command suggested by @Timur Shtatland $ conda env list -v -v -v | grep -v '^#' | perl -lane 'print $F[-1]' | xargs ls -lrt1d $ miniconda3/envs/opencv2_imgproc $ miniconda3/envs/GDAL_OSGEO_env 

Answer: Conda history files Aside from file/folder dates, Conda also records the history of all environment changes in the conda-meta/history file relative to each environment folder, so that could also be consulted. All entries begin with a date stamp (==> YYYY-MM-DD HH:MM:SS <==), so assuming the first entry corresponds with env creation, one could do something like #!/bin/bash for env_hist in path/to/envs/*/conda-meta/history; do env_prefix=$(dirname $(dirname $env_hist)) echo "$(head -n1 $env_hist) $env_prefix" done | sort  to print something like ==> 2020-09-28 16:12:49 <== path/to/envs/pymc39 ==> 2020-11-08 18:15:26 <== path/to/envs/bioc_3_12 ==> 2020-11-22 17:19:08 <== path/to/envs/snakemake_5_29 ==> 2021-01-23 00:08:33 <== path/to/envs/pymc3_11 ==> 2021-01-23 00:12:53 <== path/to/envs/jupyter ==> 2021-03-09 22:50:38 <== path/to/envs/multiqc ==> 2021-03-24 13:20:07 <== path/to/envs/grayskull ==> 2021-04-05 23:40:01 <== path/to/envs/snakemake_6_1
Question: Currently, I am building the async frontend to my TF2 model. Now it works as two services, 1st service is a twisted service, and 2nd service is a TensorFlow serving. The async web client is being used to query the model asynchronously. For practical reasons, I've deployed the model into the GCP AI Platform, and I can get data from it using the python code from examples, and it is okay. But the thing is that the Google API client is synchronous, and I would like to use the asynchronous client. Since, AFAIK, there are no actively supported async clients for GCP, I tried to get straightforward and use REST. The model input is the same on TensorFlow serving (GCP AI Platform uses TensorFlow serving internally, I believe). To perform the async call, I need to have:  Model URL. (I have it) Input data. (I also have it) Access token.  I saw some examples that are: import googleapiclient.discovery credentials = service_account.Credentials.from_service_account_file( '/path/to/key.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])  But the issue is that credential.token is None, so I can't use it. So I have a question: how could I get the access token to use in the rest request then? Or maybe there is another but better way of doing that? I already saw the following question: How to get access token from instance of google.oauth2.service_account.Credentials object? but I am think that it is slightly irrelevant.

Answer: The following code sets up the data structures for managing credentials (OAuth tokens) from a service account. No tokens are requested at this point. credentials = service_account.Credentials.from_service_account_file( '/path/to/key.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])  Tokens are not requested from the Google auth server until required. There are several reasons: a) network calls take time - a significant amount of time for network failures; b) tokens expire; c) tokens are cached until they (almost) expire. To generate a token, call the refresh() method: import google.auth.transport.requests request = google.auth.transport.requests.Request() credentials.refresh(request)  credential.token will now contain an OAuth Access Token else an exception will be thrown (network error, etc.).
Question: I have a 2D pytorch tensor of shape n by m. I want to index the second dimension using a list of indices (which could be done with torch.gather) then then also set new values to the result of the indexing. Example: data = torch.tensor([[0,1,2], [3,4,5], [6,7,8]]) # shape (3,3) indices = torch.tensor([1,2,1], dtype=torch.long).unsqueeze(-1) # shape (3,1) # data tensor: # tensor([[0, 1, 2], # [3, 4, 5], # [6, 7, 8]])  I want to select the specified indices per row (which would be [1,5,7] but then also set these values to another number - e.g. 42 I can select the desired columns row wise by doing: data.gather(1, indices) tensor([[1], [5], [7]]) data.gather(1, indices)[:] = 42 # **This does NOT work**, since the result of gather # does not use the same storage as the original tensor  which is fine, but I would like to change these values now, and have the change also affect the data tensor. I can do what I want to achieve using this, but it seems to be very un-pythonic: max_index = torch.max(indices) for i in range(0, max_index + 1): mask = (indices == i).nonzero(as_tuple=True)[0] data[mask, i] = 42 print(data) # tensor([[ 0, 42, 2], # [ 3, 4, 42], # [ 6, 42, 8]])  Any hints on how to do that more elegantly?

Answer: What you are looking for is torch.scatter_ with the value option.  Tensor.scatter_(dim, index, src, reduce=None) → Tensor Writes all values from the tensor src into self at the indices specified in the index tensor. For each value in src, its output index is specified by its index in src for dimension != dim and by the corresponding value in index for dimension = dim. With 2D tensors as input and dim=1, the operation is: self[i][index[i][j]] = src[i][j]  No mention of the value parameter though...  With value=42, and dim=1, this will have the following effect on data: data[i][index[i][j]] = 42  Here applied in-place: >>> data.scatter_(index=indices, dim=1, value=42) >>> data tensor([[ 0, 42, 2], [ 3, 4, 42], [ 6, 42, 8]])
Question: I have a dataframe where the index is the first date of each month and the size column is the frequency for that month, e.g.  Using .index on the dataframe confirms the type of the index is DatetimeIndex: DatetimeIndex(['2006-12-01', ...], dtype='datetime64[ns]', name='created_at_month', length=175, freq=None)  Using .plot() on the DataFrame I can produce a line graph per month:  However, it only lists every other year on the x axis, and I'd like it to list each year on the axis. I would expect to be able to do ax.xaxis.set_major_locator(mdates.YearLocator(1)) ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y'))  However this doesn't output any labels at all. If I add a minor formatter (ax.xaxis.set_minor_formatter(mdates.DateFormatter('%d %m %Y'))), I get this:  What am I doing wrong here to cause the dates to change? The relevant versions are:  Matplotlib: 3.3.4 Pandas: 1.2.4 Python: 3.8.8 

Answer: As reported here, for some reason pandas's plot shows this issue. You can overcome this issue by replacing pandas' plot with matplotlib.pyplot.plot. You can take this answer as a reference for 2 datetime ticks on x axis (month and year, or month and day, or day and hour, as you need), using two different axis in palce of minor and major ticks. import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import numpy as np df = pd.DataFrame({'created_at_month': pd.date_range(start = '2006-12-01', end = '2020-12-01', freq = 'MS')}) df['size'] = np.random.randint(0, 200, len(df)) df = df.set_index('created_at_month') fig, ax = plt.subplots() ax.plot(df.index, df['size']) ax.xaxis.set_major_locator(mdates.YearLocator(base = 1, month = 1, day = 1)) ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y')) plt.show()
Question: From ThreadPoolExecutor  Changed in version 3.8: Default value of max_workers is changed to min(32, os.cpu_count() + 4). This default value preserves at least 5 workers for I/O bound tasks. It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL. And it avoids using very large resources implicitly on many-core machines.  According to my understanding of GIL, thread based concurrency is only possible for I/O bound tasks. For CPU bound tasks, thread based concurrency is NOT possible, meaning for CPU bound tasks, GIL forces only single threaded execution. My understanding appears to contradict the bolded line in the ThreadPoolExecutor. What am I misunderstanding here? Furthermore, what does  which release the GIL  mean? Don't CPU bound tasks keep hold of the GIL (unless it is preempted)? From this answer, I suspect this has something to do with  spending most of its time in an external library designed to release the GIL (like NumPy)  Does that mean thread based concurrency for CPU bound tasks is actually possible provided that threads are doing the CPU bound tasks within a some specially designed external library "designed to release the GIL"?

Answer: Yes, exactly. Since the GIL protects python interpreter state, a library can release the lock if it has a significant amount of work to do that doesn't involve accessing Python variables or calling Python functions. NumPy is one such library that can frequently do this.
Question: According to this firmware post, the HP Prime graphing calculator supports Python. However, I cannot find any guide as to how to run python files in the calculator (even within HP's own 700 page long user manual). Does anyone know how to execute these files?  For reference, I have HP Prime's connectivity kit (CK) installed, so I am somewhat able to transfer python code (by copy-pasting into CK's "Programs" section). However, I think it's reading it as Prime Programming Language instead, as it does not run.  Edit: HP Prime has rebooted and now there is a Python app, allowing me to run some files. Unfortunately, I cannot access any Python libraries. That is, I can only run files that do not have "import _____" in them. This seems like a problem; anyone know how to resolve? Also a further problem is that the files are not actually saved in my calculator, as far as I can tell.

Answer: I couldn't really find a good source to read about Python support in this particular brand, but in general, graphing calculators have much more limited memory than personal computers, so they do not choose CPython or any of the heftier implementations of the Python language. They will instead use lightweight implementations like MicroPython or CircuitPython (not these ones exactly but maybe a derivation). These implementations don't have the full standard library of CPython and can have different modules particular to their intended contexts. There probably ARE some modules you can use, but without proper documentation it's hard to say which. It may go without saying but you are certainly restricted from downloading arbitrary Python libraries.
Question: I have 2 csv files, my codes are as below. df = pd.read_csv("test.csv", sep='\t',skiprows=range(9),names=['A', 'B', 'C','D']) df2 = pd.read_csv("LoadMatch_Limit.csv",skiprows=range(1),names=['X','Y']) fig = px.line([df,df2], x=['A','X'] , y=['D','Y'])  I would like my line chart, x-axis to take from (columns 'A' and 'X') and my y-axis to take from (columns 'D' and 'Y'). Is there anyway I can plot these 2 charts as one figure?

Answer: You could create the two plots and combine them with plotly graph objects import plotly.express as px import plotly.graph_objects as go fig1 = px.line(df, x='A', y='D') fig2 = px.line(df2, x='X', y='Y') fig = go.Figure(data = fig1.data + fig2.data) fig.show()
Question: I am fine-tuning a BERT model for a multiclass classification task. My problem is that I don't know how to add "early stopping" to those Trainer instances. Any ideas?

Answer: There are a couple of modifications you need to perform, prior to correctly using the EarlyStoppingCallback(). from transformers import EarlyStoppingCallback, IntervalStrategy ... ... # Defining the TrainingArguments() arguments args = TrainingArguments( output_dir = "training_with_callbacks", evaluation_strategy = IntervalStrategy.STEPS, # "steps" eval_steps = 50, # Evaluation and Save happens every 50 steps save_total_limit = 5, # Only last 5 models are saved. Older ones are deleted. learning_rate=2e-5, per_device_train_batch_size=batch_size, per_device_eval_batch_size=batch_size, num_train_epochs=5, weight_decay=0.01, push_to_hub=False, metric_for_best_model = 'f1', load_best_model_at_end=True)  You need to:  Use load_best_model_at_end = True (EarlyStoppingCallback() requires this to be True). evaluation_strategy = 'steps' or IntervalStrategy.STEPS instead of 'epoch'. eval_steps = 50 (evaluate the metrics after N steps). metric_for_best_model = 'f1'  In your Trainer(): trainer = Trainer( model, args, ... compute_metrics=compute_metrics, callbacks = [EarlyStoppingCallback(early_stopping_patience=3)] )  Of course, when you use compute_metrics(), for example it can be a function like: def compute_metrics(p): pred, labels = p pred = np.argmax(pred, axis=1) accuracy = accuracy_score(y_true=labels, y_pred=pred) recall = recall_score(y_true=labels, y_pred=pred) precision = precision_score(y_true=labels, y_pred=pred) f1 = f1_score(y_true=labels, y_pred=pred) return {"accuracy": accuracy, "precision": precision, "recall": recall, "f1": f1}  The return of the compute_metrics() should be a dictionary and you can access whatever metric you want/compute inside the function and return. Note: In newer transformers version, the usage of Enum IntervalStrategy.steps is recommended (see TrainingArguments()) instead of plain steps string, the latter being soon subject to deprecation.