eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I created widgets in a grid-layout. The widgets are stretching based on the window. Is it possible to avoid the stretching and align them as shown in picture below? I created a code to achieve this, but I feel it is not a straightforward solution. If there are any better solutions to achieve this, please share them. Grid layout result: from PyQt5.QtWidgets import * app =QApplication([]) window=QWidget() GL=QGridLayout(window) GL.addWidget(QPushButton('R1C1'),0,0) GL.addWidget(QPushButton('R1C2'),0,1) GL.addWidget(QPushButton('R2C1'),1,0) GL.addWidget(QPushButton('R1C1'),1,1) window.showMaximized() app.exec_()   Required Result:  My code: from PyQt5.QtWidgets import * app =QApplication([]) window=QWidget() VL=QVBoxLayout(window);HL=QHBoxLayout();VL.addLayout(HL) GL=QGridLayout();HL.addLayout(GL) GL.addWidget(QPushButton('R1C1'),0,0) GL.addWidget(QPushButton('R1C2'),0,1) GL.addWidget(QPushButton('R2C1'),1,0) GL.addWidget(QPushButton('R1C1'),1,1) HL.addStretch();VL.addStretch() window.showMaximized() app.exec_() 

Answer: The QGridLaout class doesn't have any simple convenience methods like QBoxLayout.addStretch() to do this. But the same effect can be achieved by adding some empty, stretchable rows/columns, like this: GL.setRowStretch(GL.rowCount(), 1) GL.setColumnStretch(GL.columnCount(), 1)
Question: newbie in Python here when I run this simple code (to load Harry Potter page and simply print it) it returns me Error with the wrong name I wanted to search (harry plotter) can anyone tell me how to fix? thank you! import wikipedia page = wikipedia.page("Harry Potter") print(page.summary)  Error message: Traceback (most recent call last): File "C:\Users\Lidor\PycharmProjects\pythonProject\search_engine.py", line 7, in <module> page = wikipedia.page(search[0]) File "C:\Users\Lidor\PycharmProjects\pythonProject\venv\lib\site- packages\wikipedia\wikipedia.py", line 276, in page return WikipediaPage(title, redirect=redirect, preload=preload) File "C:\Users\Lidor\PycharmProjects\pythonProject\venv\lib\site- packages\wikipedia\wikipedia.py", line 299, in __init__ self.__load(redirect=redirect, preload=preload) File "C:\Users\Lidor\PycharmProjects\pythonProject\venv\lib\site- packages\wikipedia\wikipedia.py", line 345, in __load raise PageError(self.title) wikipedia.exceptions.PageError: Page id "harry plotter" does not match any pages. Try another id! 

Answer: This appears to be a strange result of auto_suggest being set by default. If you do wikipedia.page("Harry Potter", auto_suggest=False)  It works fine. Otherwise it autocompletes potter to plotter, hence the error.
Question: I want to use some pandas style resources and I want to hide table indexes on streamlit. I tryed this: import streamlit as st import pandas as pd table1 = pd.DataFrame({'N':[10, 20, 30], 'mean':[4.1, 5.6, 6.3]}) st.dataframe(table1.style.hide_index().format(subset=['mean'], decimal=',', precision=2).bar(subset=['mean'], align="mid"))  but regardless the .hide_index() I got this:  Ideas to solve this?

Answer: Documentation for st.dataframe shows "Styler support is experimental!" and maybe this is the problem. But I can get table without index if I use .to_html() and st.write() import streamlit as st import pandas as pd df = pd.DataFrame({'N':[10, 20, 30], 'mean':[4.1, 5.6, 6.3]}) styler = df.style.hide_index().format(subset=['mean'], decimal=',', precision=2).bar(subset=['mean'], align="mid") st.write(styler.to_html(), unsafe_allow_html=True) #st.write(df.to_html(index=False), unsafe_allow_html=True)
Question: Below is my DF df= pd.DataFrame({'col1': ['[7]', '[30]', '[0]', '[7]'], 'col2': ['[0%, 7%]', '[30%]', '[30%, 7%]', '[7%]']}) col1 col2 [7] [0%, 7%] [30] [30%] [0] [30%, 7%] [7] [7%]  The aim is to check if col1 value is contained in col2 below is what I've tried df['test'] = df.apply(lambda x: str(x.col1) in str(x.col2), axis=1)  Below is the expected output col1 col2 col3 [7] [0%, 7%] True [30] [30%] True [0] [30%, 7%] False [7] [7%] True 

Answer: You can also replace the square brackets with word boundaries \b and use re.search like in import re #... df.apply(lambda x: bool(re.search(x['col1'].replace("[",r"\b").replace("]",r"\b"), x['col2'])), axis=1) # => 0 True # 1 True # 2 False # 3 True # dtype: bool  This will work because \b7\b will find a match in [0%, 7%] as 7 is neither preceded nor followed with letters, digits or underscores. There won't be any match found in [30%, 7%] as \b0\b does not match a zero after a digit (here, 3).
Question: I'm learning how to use airflow to build machine learning pipeline. But didn't find a way to pass pandas dataframe generated from 1 task into another task... It seems that need to convert the data to JSON format or save the data in database within each task? Finally, I had to put everything in 1 task... Is there anyway to pass dataframe between airflow tasks? Here's my code: from datetime import datetime import pandas as pd import numpy as np import os import lightgbm as lgb from sklearn.model_selection import train_test_split from sklearn.model_selection import StratifiedKFold from sklearn.metrics import balanced_accuracy_score from airflow.decorators import dag, task from airflow.operators.python_operator import PythonOperator @dag(dag_id='super_mini_pipeline', schedule_interval=None, start_date=datetime(2021, 11, 5), catchup=False, tags=['ml_pipeline']) def baseline_pipeline(): def all_in_one(label): path_to_csv = os.path.join('~/airflow/data','leaf.csv') df = pd.read_csv(path_to_csv) y = df[label] X = df.drop(label, axis=1) folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=10) lgbm = lgb.LGBMClassifier(objective='multiclass', random_state=10) metrics_lst = [] for train_idx, val_idx in folds.split(X, y): X_train, y_train = X.iloc[train_idx], y.iloc[train_idx] X_val, y_val = X.iloc[val_idx], y.iloc[val_idx] lgbm.fit(X_train, y_train) y_pred = lgbm.predict(X_val) cv_balanced_accuracy = balanced_accuracy_score(y_val, y_pred) metrics_lst.append(cv_balanced_accuracy) avg_performance = np.mean(metrics_lst) print(f"Avg Performance: {avg_performance}") all_in_one_task = PythonOperator(task_id='all_in_one_task', python_callable=all_in_one, op_kwargs={'label':'species'}) all_in_one_task # dag invocation pipeline_dag = baseline_pipeline() 

Answer: Although it is used in many ETL tasks, Airflow is not the right choice for that kind of operations, it is intended for workflow not dataflow. But there are many ways to do that without passing the whole dataframe between tasks. You can pass information about the data using xcom.push and xcom.pull: a. Save the outcome of the first task somewhere (json, csv, etc.) b. Pass to xcom.push information about saved file. E.g. file name, path. c. Read this filename using xcom.pull from the other task and perform needed operation. Or: Everything above using some database tables: a. In task_1 you can download data from table_1 in some dataframe, process it and save in another table_2 (df.to_sql()). b. Pass the name of the table using xcom.push. c. From the other task get table_2 using xcom.pull and read it with df.read_sql(). Information on how to use xcom you can get from airflow examples. Example: https://github.com/apache/airflow/blob/main/airflow/example_dags/tutorial_etl_dag.py IMHO there are many other better ways, I have just written what I tried.
Question: I have following multi-dimensional array: windows = array([[[[[[0., 0.], [1., 0.]], [[0., 0.], [1., 0.]], [[0., 0.], [1., 0.]]], [[[0., 1.], [0., 0.]], [[0., 1.], [0., 0.]], [[1., 0.], [0., 0.]]], [[[1., 0.], [0., 0.]], [[0., 1.], [0., 0.]], [[0., 1.], [0., 0.]]]]]]) print(windows.shape) (1, 1, 3, 3, 2, 2) # (n, d, a, b, c, c) w = a * (c*c), h = b * (c*c)  I want to get next resulting array: mask = array([[ [[0., 0., 0., 0., 0., 0.], [1., 0., 1., 0., 1., 0.], [0., 1., 0., 1., 1., 0.], [0., 0., 0., 0., 0., 0.], [1., 0., 0., 1., 0., 1.], [0., 0., 0., 0., 0., 0.]] ]], dtype=np.float32) print(mask.shape) (1, 1, 6, 6) # (n, d, w, h)  Basically, I want to squeeze last 4 dimensions into 2-d matrix so the final shape become (n, d, w, h), in this case (1, 1, 6, 6). I tried np.concatenate(windows, axis = 2), but it didnt concatenate along 2nd dimension and reduced for some reason first(although I set axis = 2) 'n' dimension. Additional information:  windows is a result of following code snippet windows = np.lib.stride_tricks.sliding_window_view(arr, (c, c), axis (-2,-1), writeable = True) # arr.shape == mask.shape windows = windows[:, :, ::c, ::c] # these are non-overlapping windows of arr with size (c,c) windows = ... # some modifications of windows  Now I want to build from these windows array with shape of arr.shape, this array called mask in example above. Simple reshape doesn't work because it returns elements in wrong order.

Answer: IIUC, you want to merge dimensions 2+4 and 3+5, an easy way would be to swapaxes 4 and 5 (or -3 and -2), and reshape to (1,1,6,6): windows.swapaxes(-2,-3).reshape(1,1,6,6)  output: array([[[[0., 0., 0., 0., 0., 0.], [1., 0., 1., 0., 1., 0.], [0., 1., 0., 1., 1., 0.], [0., 0., 0., 0., 0., 0.], [1., 0., 0., 1., 0., 1.], [0., 0., 0., 0., 0., 0.]]]])
Question: I have the following code: import matplotlib.pyplot as plt import numpy as np import pandas as pd data = pd.read_csv("Ari_atlag.txt", sep = '\t', header = 0) #Num_array = pd.DataFrame(data).to_numpy() print(data.head()) data.plot() #data.columns = ['Date', 'Number_of_test', 'Avarage_of_ARI'] #print(Num_array) plt.show()  Output:  Date Number_of_test Avarage_of_ARI 0 2011-01 22 0.568734 1 2011-02 5 0.662637 2 2011-03 0 0.000000 3 2011-04 3 0.307692 4 2011-05 6 0.773611 Process finished with exit code 0  and the plot. But with this code in the plot, the x axis is the index. But I want to get Date in x axis. How to plot the Date with Number_of_test and Date with Avarage_of_ARI I think, I somehow should change string(date) to dates, but don't know how to do that. Best

Answer: Use x='Date' as parameter of plot: df.plot(x='Date') plt.show()
Question: I found this post, but it's not quite my scenario. Is there an efficient way of comparing two data frames The reason I want to compare two dataframes is that I am looking for changes that may have occured. (Think "audit"). The two frames have exactly the same column layout, just that one may have more or less rows than the other, and values may have changed in two of the columns.  ID Period Price OtherID 0 10100001 2019-10-01 8995.00 ABBD 1 38730001 2019-11-01 38227.71 EIRU 2 30100453 2019-12-01 22307.00 DDHF 3 92835543 2020-01-01 2310.00 DDGF 4 66453422 2020-02-01 12113.29 DGFH ID Period Price OtherID 0 10100001 2019-10-01 5.00 ABBD 1 38730001 2019-11-01 38227.71 XXXX 2 30100453 2019-12-01 22307.00 DDHF 3 92835543 2020-01-01 2310.00 DDGF 4 66453422 2020-02-01 12113.29 DGFH 5 22223422 2020-02-01 123.29 HHYG  The two columns that I am suspicious about are "Price" and "OtherID". I want to find any changes in the Price or OtherID. I see three scenarios: 1. row has been added 2. row has been deleted 3. row doesn't match  I can iterate through it all, but I'm wondering if there is some pandas magic out there that will do it in one fell swoop. The output I seek is something like this:  ID Period Analysis 0 10100001 2019-10-01 Changed 1 38730001 2019-11-01 Changed 3 22223422 2020-02-01 New  And just to be clear, ID by itself is not unique. And Period by itself is also not unique. The two together are.

Answer: Use compare after merge your 2 dataframes on ID and Period: out = pd.merge(df1, df2, on=['ID', 'Period'], how='outer', suffixes=('_df1', '_df2')).set_index(['ID', 'Period']) out.columns = pd.MultiIndex.from_tuples(out.columns.str.split('_').map(tuple)) \ .swaplevel() out = out['df1'].compare(out['df2'])  Output: >>> out Price OtherID self other self other ID Period 10100001 2019-10-01 8995.0 5.00 NaN NaN 38730001 2019-11-01 NaN NaN EIRU XXXX 22223422 2020-02-01 NaN 123.29 NaN HHYG # Summarize >>> out.swaplevel(axis=1)['self'].isna().all(axis=1) \ .replace({True: 'New', False: 'Changed'}) ID Period 10100001 2019-10-01 Changed 38730001 2019-11-01 Changed 22223422 2020-02-01 New dtype: object  If you append keep_shape=True parameter: >>> out['df1'].compare(out['df2'], keep_shape=True) Price OtherID self other self other ID Period 10100001 2019-10-01 8995.0 5.00 NaN NaN 38730001 2019-11-01 NaN NaN EIRU XXXX 30100453 2019-12-01 NaN NaN NaN NaN 92835543 2020-01-01 NaN NaN NaN NaN 66453422 2020-02-01 NaN NaN NaN NaN 22223422 2020-02-01 NaN 123.29 NaN HHYG
Question: I want to build a model with several Conv1d layers followed by several Linear layers. Conv1d layers will work for data of any given length, the problem comes at the first Linear layer, because the data length is unknown at initialization time. Every time the length of the input data changes, the output size of Conv1d layers will change, hence a change in the required in_features of the first Linear layer. Note: I learned CNN and I am aware of how to calculate the output dimensions by hand. I am looking for a programmatic way to determine it, because I have to experiment many times with different length of input data. Question: In pytorch, how do you automatically figure out the output dimension after many Conv1d layers and set the in_features for the following Linear layer?

Answer: You can use the builtin nn.LazyLinear which will find the in_features on the first inference and initialize the appropriate number of weights accordingly: linear = nn.LazyLinear(out_features)
Question: I have implemented yt-dlp as part of my Python script, it works well, but I am unable to get the rate-limit feature to work. If you run the same command from the CLI the rate is limited correctly, is anyone able to tell me the correct syntax? I have tried several combinations such as rate-limit, limit-rate 0.5m, 500k, 500KiB, 500, and none seem to work  ydl_opts = { 'limit-rate': '500k', } with yt_dlp.YoutubeDL(ydl_opts) as ydl: ydl.download([link])  I am using the docs here; https://github.com/yt-dlp/yt-dlp But am confused as the CLI command works but not the embedded script version, I also tried replacing - with _ but still to no effect, do you have any ideas? Other options in the ydl_opts work without issue Hopefully we can resolve the correct syntax rather than having to implement Trickle or throttling the socket.

Answer: Looking at the source code you'll find that the option you're looking for is called ratelimit. Its value should be a float: ydl_opts = { 'ratelimit': 500000 } with yt_dlp.YoutubeDL(params=ydl_opts) as ydl: ydl.download([link])
Question: I recently started working on Machine Learning with Linear Regression. I have used a LinearRegression (lr) to predict some values. Indeed, my predictions were bad, and I was asked to change the hyperparameters to obtain better results. I used the following command to obtain the hyperparameters: lr.get_params().keys() lr.get_params()  and obtained the following:  'copy_X': True, 'fit_intercept': True, 'n_jobs': None, 'normalize': False, 'positive': False}  and dict_keys(['copy_X', 'fit_intercept', 'n_jobs', 'normalize', 'positive'])  Now, this is where issues started to raise. I have tried to find the correct syntax to use the .set_params() function, but every answer seemed outside my comprehension. I have tried to assign a positional arguments since commands such as lr.set_params('normalize'==True) returned TypeError: set_params() takes 1 positional argument but 2 were given  and lr.set_params(some_params = {'normalize'}) returned ValueError (`ValueError: Invalid parameter some_params for estimator LinearRegression(). Check the list of available parameters with estimator.get_params().keys().  Can someone provide a simple explanation of how this function works?

Answer: The correct syntax is set_params(**params) where params is a dictionary containing the estimator's parameters, see the scikit-learn documentation. from sklearn.linear_model import LinearRegression reg = LinearRegression() reg.get_params() # {'copy_X': True, # 'fit_intercept': True, # 'n_jobs': None, # 'normalize': False, # 'positive': False} reg.set_params(**{ 'copy_X': False, 'fit_intercept': False, 'n_jobs': -1, 'normalize': True, 'positive': True }) reg.get_params() # {'copy_X': False, # 'fit_intercept': False, # 'n_jobs': -1, # 'normalize': True, # 'positive': True}
Question: Say I have a descending ordered list in terms of preference: person_choice = ['amy', 'bob', 'chad', 'dan', 'emily']  meaning, amy is preferred to bob, who is preferred to chad and so on. And I have a couple more lists like these (with names appearing in no particular order and list length): group1 = ['joyce', 'amy', 'emily', 'karen', 'rebecca'] group2 = ['chad', 'kyle', 'michael', 'neo', 'bob'] ...  In each of the group1 and group2, I would like to select the person within this list that appears in person_choice with the highest preference. So, an expected output is like this: group1_choice = 'amy' group2_choice = 'bob'  Assuming that for each groupX, there is at least one person appearing in person_choice, is there a one-liner to do this in Python without complicated for loop?

Answer: Using max with a custom key: person_choice = ['amy', 'bob', 'chad', 'dan', 'emily'] group1 = ['joyce', 'amy', 'emily', 'karen', 'rebecca'] group2 = ['chad', 'kyle', 'michael', 'neo', 'bob'] def choice(group): return max(group, key=lambda name: -person_choice.index(name) if name in person_choice else float('-inf')) print(choice(group1)) print(choice(group2))  Result: amy bob
Question: I'm learning about classes and don't understand this: class MyClass: var = 1 one = MyClass() two = MyClass() print(one.var, two.var) # out: 1 1 one.var = 2 print(one.var, two.var) # out: 2 1  I thought that class variables are accessible by all instances, why is the Class variable not updating for all its instances?

Answer: It doesn't change for all of them because doing this: one.var = 2, creates a new instance variable with the same name as the class variable, but only for the instance one. After that, one will first find its instance variable and return that, while two will only find the class variable and return that. To change the class variable I suggest two options:  create a class method to change the class variable (my preference)  change it by using the class directly   class MyClass: var = 1 @classmethod def change_var(cls, var): cls.var = var one = MyClass() two = MyClass() print(one.var, two.var) # out: 1 1 one.change_var(2) # option 1 print(one.var, two.var) # out: 2 2 MyClass.var = 3 # option 2 print(one.var, two.var) # out: 3 3
Question: If I send a request to this API: from fastapi import FastAPI from pydantic import BaseModel app = FastAPI() class Response(BaseModel): var_name: str @app.put("/", response_model=Response) def simple_server(a: str): response = Response(var_name=a) return response  I get a response which this json file {"var_name1": "a"}. In addition, I get a very beautiful Swagger UI that illustrate the fields of response. My question is, how can I get this json file {"var-name1": "a"} (this is with a hyphen instead of an underscore) with the same nice typing in Swagger docs? Obviously, I cannot name the var_name attribute var-name in Response dataclass.

Answer: Modify your pydantic object slightly: from pydantic import BaseModel, Field class Response(BaseModel): var_name: str = Field(alias="var-name") class Config: allow_population_by_field_name = True  The allow_population_by_field_name option is needed to allow creating object with field name, without it you could instantiate it only with alias name.
Question: I have a dataframe where the indexes are not numbers but strings (specifically, name of countries) and they are all unique. Given the name of a country, how do I find its row number (the 'number' value of the index)? I tried df[df.index == 'country_name'].index but this doesn't work.

Answer: We can use Index.get_indexer: df.index.get_indexer(['Peru']) [3]  Or we can build a RangeIndex based on the size of the DataFrame then subset that instead: pd.RangeIndex(len(df))[df.index == 'Peru'] Int64Index([3], dtype='int64')   Since we're only looking for a single label and the indexes are "all unique" we can also use Index.get_loc: df.index.get_loc('Peru') 3   Sample DataFrame: import pandas as pd df = pd.DataFrame({ 'A': [1, 2, 3, 4, 5] }, index=['Bahamas', 'Cameroon', 'Ecuador', 'Peru', 'Japan'])  df:  A Bahamas 1 Cameroon 2 Ecuador 3 Peru 4 Japan 5
Question: I have a dataset that looks like this (assume this has 4 categories in Clicked, the head(10) only showed 2 categories):  Rank Clicked 0 2.0 Cat4 1 2.0 Cat4 2 2.0 Cat4 3 1.0 Cat1 4 1.0 Cat4 5 2.0 Cat4 6 2.0 Cat4 7 3.0 Cat4 8 5.0 Cat4 9 5.0 Cat4  This is a code that returns this plot: eee = (df.groupby(['Rank','Clicked'])['Clicked'].count()/df.groupby(['Rank'])['Clicked'].count()) eee.unstack().plot.bar(stacked=True) plt.legend(['Cat1','Cat2','Cat3','Cat4']) plt.xlabel('Rank')   Is there a way to achieve this with seaborn (or matplotlib) instead of the pandas plotting capabilities? I tried a few ways, both of running the seaborn code and of preprocessing the dataset so it's on the correct format, with no luck.

Answer: Seaborn doesn't support stacked barplot, so you need to plot the cumsum: # calculate the distribution of `Clicked` per `Rank` distribution = pd.crosstab(df.Rank, df.Clicked, normalize='index') # plot the cumsum, with reverse hue order sns.barplot(data=distribution.cumsum(axis=1).stack().reset_index(name='Dist'), x='Rank', y='Dist', hue='Clicked', hue_order = distribution.columns[::-1], # reverse hue order so that the taller bars got plotted first dodge=False)  Output:  Preferably, you can also reverse the cumsum direction, then you don't need to reverse hue order: sns.barplot(data=distribution.iloc[:,::-1].cumsum(axis=1) # we reverse cumsum direction here .stack().reset_index(name='Dist'), x='Rank', y='Dist', hue='Clicked', hue_order=distribution.columns, # forward order dodge=False)  Output:
Question: I'm using python 3.8.0 for my FastAPI app. It uses the .env file located on the root of a project directory. I am using the dotenv package, and the location of the .env file is hardcoded within the app. Here is my unit file [Unit] Description=Gunicorn instance for my_app After=network.target [Service] User=nginx Group=nginx WorkingDirectory=/usr/share/nginx/html/my_app/ Environment="PATH=/usr/share/nginx/html/my_app/venv/bin" ExecStart=/usr/share/nginx/html/my_app/venv/bin/gunicorn --bind unix:/usr/share/nginx/html/my_app/my_app.sock -w 4 -k uvicorn.workers.UvicornWorker app.main:app [Install] WantedBy=multi-user.target  The challenge is to run two versions (production and test) of the same app using two different .env on two different ports. I'll have to create a second unit file. But how can I pass the name of two diffeent env file names to the app for further usage. These files contain database connections, etc. I imagine it roughly like this 1st unit file ExecStart=/usr/share/nginx/html/my_app/venv/bin/gunicorn --bind unix:/usr/share/nginx/html/my_app/my_app.sock -w 4 -k uvicorn.workers.UvicornWorker app.main:app --env_file_name=".env.prod"  2nd unit file ExecStart=/usr/share/nginx/html/my_app/venv/bin/gunicorn --bind unix:/usr/share/nginx/html/my_app/my_app.sock -w 4 -k uvicorn.workers.UvicornWorker app.main:app --env_file_name=".env.dev" 

Answer: You can set the path from which systemd will read the environment for your process exactly in unit file configuration. The setting is called EnvironmnetFile=. Just set the option to the path for .env.prod in one unit file and for the path to .env.test for another.