eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to deprecate an old syntax for a function in a Python library. In order to effectively detect whether someone is using the old syntax, I need to know whether an argument is called positionally or through a keyword. Is there a way to detect this? As an example, consider this function: def store(name='', value=0): # Some functionality here...  Can this function know whether it has been called like this: store('ben', 5)  or like this? store(name='ben', value=5) 

Answer: You could add *args to your function's arguments and check if that contains any arguments - if yes, the user passed positional arguments to your function that should have been passed as keyword arguments: def store(*args, name='', value=0): if args: # args is not empty - user passed deprecated positional arguments print(f"Warning: you passed the arguments {' and '.join(map(str, args))} as positional arguments.") print("This is deprecated - please pass them as keyword arguments") name = args[0] if len(args) >= 2: value = args[1] store('ben', 5) store(name='ben', value=5)  Tested in the interactive Python console: >>> store('ben', 5) Warning: you passed the arguments ben and 5 as positional arguments. This is deprecated - please pass them as keyword arguments >>> store('ben', value=5) Warning: you passed the arguments ben as positional arguments. This is deprecated - please pass them as keyword arguments >>> store(name='ben', value=5) >>>
Question: While following the Jupyter notebooks for the course I hit upon an error when these lines are run. I know that the cnn_learner line has got no errors whatsoever, The problem lies in the lr_find() part It seems that learn.lr_find() does not want to return two values! Although its documentation says that it returns a tuple. That is my problem. These are the lines of code: learn = cnn_learner(dls, resnet34, metrics=error_rate) lr_min,lr_steep = learn.lr_find()  The error says: not enough values to unpack (expected 2, got 1)  for the second line. Also, I get this graph with one 'marker' which I suppose is either one of the values of lr_min or lr_steep This is the graph When I run learn.lr_find() only, i.e. do not capture the output in lr_min, lr_steep; it runs well but then I do not get the min and steep learning rates (which is really important for me) I read through what lr_find does and it is clear that it returns a tuple. Its docstring says  Launch a mock training to find a good learning rate and return suggestions based on suggest_funcs as a named tuple  I had duplicated the original notebook, and when I hit this error, I ran the original notebook, with the same results. I update the notebooks as well, but no change! Wherever I have searched for this online, any sort of error hasn't popped up. The only relevant thing I found is that lr_find() returns different results of the learning rates after every run, which is perfectly fine.

Answer: I was having the same problem and I found that the lr_find() output's has updated. You can substitute the second line to lrs = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide)), and then you just substitute where you using lr_min and lr_steep to lrs.minimum and lrs.steep respectively, this should work fine and solve your problem. If you wanna read more about it, you can see this post that is in the fastai's forum.
Question: I created two dataframes (df1 and df2) in two jupyter notebooks (N1 and N2) respectively. On day 1, I used the below store command to use the df1 and its variables in N2 jupyter notebook %store -r df1  But on day 25, I created a new jupyter notebook N3 and used the below store command again %store -r df1  And it seemed to easily pull all the details of dataframe df1 into N3 jupyter notebook easily? How does this work? Aren't they valid only for that specific jupyter notebook session? Then instead of storing all dataframes as files, can we just execute a store command and store/retrieve them easily anytime?

Answer: Storemagic is an IPython feature that "Stores variables, aliases and macros in IPython’s database". Because it is an IPython feature rather than exclusive to Jupyter you can store and restore variables across many IPython and Jupyter sessions. In my environment (IPython 7.19.0) the variables get stored in the directory: $HOME/.ipython/profile_default/db/autorestore  They are stored one per file when stored using %store <name>. The files themselves are the pickled representation of the stored variables. You could manually load the variables by using the following: import pickle # Name of the previously stored variable stored_var = 'test' # myvar will contain the variable previously stored with "%store test" myvar_filename = get_ipython().ipython_dir + '/profile_default/db/autorestore/' + stored_var with open(myvar_filename, 'rb') as f: myvar = pickle.load(f)
Question: I'm new to Python. I want to check if the given list A contains any character among ('0', '2', '4', '6', '8') or not, where '0' <= A[i] <= '9'. I can do this as: if not ('0' in A or '2' in A or '4' in A or '6' in A or '8' in A): return False  but, is there any shorter way to do this? Thanks.

Answer: You can use any with generator expression A = [...] chars = ('0', '2', '4', '6', '8') return any(c in A for c in chars)
Question: I'm a self-taught Python user. In Python codes, model.fit(x_train, y_train, verbose=1, validation_split=0.2, shuffle=True, epochs=20000)  Then, 80% of the data is used for training and 20% is used for validation, and the epoch is repeated 20,000 times for training. And, shuffle=True  So, I think this code is a cross-validation, or more specifically, a k-divisional cross-validation with k=5. I was wondering if this is correct, because when I looked up the Keras code for k-fold cross-validation, I found some code that uses Scikit-learn's Kfold. I apologize for the rudimentary nature of this question, but I would appreciate it if you could help me.

Answer: The model first shuffles the data and then splits it to train and validation For the next epoch, the train & validation have already been defined in the first epoch, so it does not shuffle & split again, but uses the previously defined datasets. Therefore, it is a cross-validation.
Question: when I was trying to connect google firebase real time database, I got this error: ValueError: The default Firebase app already exists. This means you called initialize_app() more than once without providing an app name as the second argument. In most cases you only need to call initialize_app() once. But if you do want to initialize multiple apps, pass a second argument to initialize_app() to give each app a unique name.  Here is my code: import firebase_admin from firebase_admin import credentials from firebase_admin import db cred = credentials.Certificate('firebase-sdk.json') firebase_admin.initialize_app(cred, { 'databaseURL': 'https://test-139a6-default-rtdb.firebaseio.com/' }) 

Answer: You only need to initialize (create) the app once. When you have created the app, use get_app instead: # The default app's name is "[DEFAULT]" firebase_admin.get_app(name='[DEFAULT]')
Question: I have two numpy arrays a and b of shape [5, 5, 5] and [5, 5], respectively. For both a and b the first entry in the shape is the batch size. When I perform matrix multiplication option, I get an array of shape [5, 5, 5]. An MWE is as follows. import numpy as np a = np.ones((5, 5, 5)) b = np.random.randint(0, 10, (5, 5)) c = a @ b # c.shape is (5, 5, 5)  Suppose I were to run a loop over the batch size, i.e. a[0] @ b[0].T, it would result in an array of shape [5, 1]. Finally, if I concatenate all the results along axis 1, I would get a resultant array with shape [5, 5]. The code below better describes these lines. a = np.ones((5, 5, 5)) b = np.random.randint(0, 10, (5, 5)) c = [] for i in range(5): c.append(a[i] @ b[i].T) c = np.concatenate([d[:, None] for d in c], axis=1).T # c.shape evaluates to be (5, 5)  Can I get the above functionality without using loop? For example, PyTorch provides a function called torch.bmm to compute this. Thanks.

Answer: You can work this out using numpy einsum. c = np.einsum('BNi,Bi ->BN', a, b)  Pytorch also provides this einsum function with slight change in syntax. So you can easily work it out. It easily handles other shapes as well. Then you don't have to worry about transpose or squeeze operations. It also saves memory because no copy of existing matrices are created internally.
Question: I was doing some coding exercises and I ended up using a set of frozensets. Here is the code: cities = 4 roads = [[0, 1], [1, 2], [2, 0]] roads = set([frozenset(road) for road in roads]) output = [] for i in range(cities-1): for j in range(i+1, cities): if set([i,j]) not in roads: output.append([i,j])  As you can see, the if in the nested for tests for the presence of the set in the set of sets. However, it was my understanding that in this case, hashables need to be used with the in operator. If I replace set([i,j]) with [i,j], I do get the following error: TypeError: unhashable type: 'list'  So, here is my question: why does it work with the set, which is not (as far as I know) hashable and not with the list? Should it not also throw an error, what am I missing?

Answer: From my reading of the CPython source it appears that the test for contains checks if the key is found in the set; if not, and if the key is a set object, an attempt is made to convert the key to a frozenset, and then that key is tested. The same behavior exists for operations like remove, as seen here: >>> s = set([frozenset([1,2])]) >>> s {frozenset({1, 2})} >>> s.remove(set([1,2])) >>> s set()  The code in question in the interpreter is the set_contains() function in Objects/setobject.c.
Question: from typing import Callable def f() -> Callable: return f  How to explicitly define f's type? like Callable[[], Callable] I think it is slightly like a linked list, but I can't implement it. from typing import Union class Node: def __init__(self, val): self.val = val self.next: Union[Node, None] = None 

Answer: I think @chepner's answer is great. If you really do want to express this as a recursive Callable type, then you could restructure the function as a callable class and do something like this: from __future__ import annotations class F: def __call__(self) -> F: return self f = F()  You can test this with mypy to see that it maintains its type on future calls: g = f() h = g(1) # Too many arguments for "__call__" of "F" i = h() j = i(2) # Too many arguments for "__call__" of "F" k = j()
Question: I'm super amazed using the generator instead of list. But I can't find any solution for this question. What is the efficient way to get the first and last element from generator items? Because with list we can just do lst[0] and lst[-1] Thanks for the help. I can't provide any codes since it's clearly that's just what I want to know :)

Answer: You have to iterate through the whole thing. Say you have this generator: def foo(): yield 0 yield 1 yield 2 yield 3  The easiest way to get the first and last value would be to convert the generator into a list. Then access the values using list lookups. data = list(foo()) print(data[0], data[-1])  If you want to avoid creating a container, you could use a for-loop to exhaust the generator. gen = foo() first = last = next(gen) for last in gen: pass print(first, last)  Note: You'll want to special case this when there are no values produced by the generator.
Question: How can I rotate to 90° the team names (x-axis) on Plotly express? They are not turned in the right way. Here is my code. fig = px.bar(stacked_ratio, y="percent", x="team", color="outcome", color_discrete_map=colors, title="Long-Form Input") fig.show()  Here how it looks: 

Answer: You should be able to update your x-axis from a figure object with the update_xaxes method: fig = px.bar(stacked_ratio, y="percent", x="team", color="outcome", color_discrete_map=colors, title="Long-Form Input") fig.update_xaxes(tickangle=90) fig.show()  You can see all options for fig.update_xaxes on the plotly website here: https://plotly.com/python/reference/layout/xaxis/
Question: I want to create a simple script which will upload a file to my Drive every 5 minutes using cronjob. This is the code I have so far using the boilerplate code I extracted from different locations (mainly 2: getting started page & create page): from __future__ import print_function from apiclient import errors import pickle import os.path from googleapiclient.discovery import build from google_auth_oauthlib.flow import InstalledAppFlow from google.auth.transport.requests import Request from googleapiclient.http import MediaFileUpload def activateService(): creds = None if os.path.exists('token.pickle'): with open('token.pickle', 'rb') as token: creds = pickle.load(token) if not creds or not creds.valid: if creds and creds.expired and creds.refresh_token: creds.refresh(Request()) else: flow = InstalledAppFlow.from_client_secrets_file( 'credentials.json', SCOPES) creds = flow.run_local_server(port=0) with open('token.pickle', 'wb') as token: pickle.dump(creds, token) return build('drive', 'v3', credentials=creds) SCOPES = ['https://www.googleapis.com/auth/drive.metadata.readonly', 'https://www.googleapis.com/auth/drive.file'] myservice = activateService() file_metadata = {'name': 'myFile.txt'} media = MediaFileUpload("myFile.txt", mimetype="text/plain") file = myservice.files().create(body=file_metadata, media_body=media, fields='id').execute()  The above code creates the file successfully in the "root" location, but now how I can make it so that it overwrites the previously created file instead of creating new versions everytime? I think I need to use the update API call (https://developers.google.com/drive/api/v3/reference/files/update) but there's no example code on this documentation page which has brought me to a roadblock. Any help trying to decipher that API page to create Python code would be much appreciated, thank you!

Answer: Your code uses which will create a new file every time. myservice.files().create  You need to use File update The only diffrence is you need to pass the file id. file = service.files().update(fileId=file_id, media_body=media_body).execute()
Question: In general, we will df.drop('column_name', axis=1) to remove a column in a DataFrame. I want to add this transformer into a Pipeline Example: numerical_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean')), ('scaler', StandardScaler(with_mean=False)) ])  How can I do it?

Answer: You can encapsulate your Pipeline into a ColumnTransformer which allows you to select the data that is processed through the pipeline as follows: import pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.impute import SimpleImputer from sklearn.compose import make_column_selector, make_column_transformer col_to_exclude = 'A' df = pd.DataFrame({'A' : [ 0]*10, 'B' : [ 1]*10, 'C' : [ 2]*10}) numerical_transformer = make_pipeline SimpleImputer(strategy='mean'), StandardScaler(with_mean=False) ) transform = ColumnTransformer( (numerical_transformer, make_column_selector(pattern=f'^(?!{col_to_exclude})')) ) transform.fit_transform(df)  NOTE: I am using here a regex pattern to exclude the column A.
Question: I created some data from two superposed normal distributions and then applied sklearn.neighbors.KernelDensity and scipy.stats.gaussian_kde to estimate the density function. However, using the same bandwith (1.0) and the same kernel, both methods produce a different outcome. Can someone explain me the reason for this? Thanks for help. Below you can find the code to reproduce the issue: import matplotlib.pyplot as plt import numpy as np from scipy.stats import gaussian_kde import seaborn as sns from sklearn.neighbors import KernelDensity n = 10000 dist_frac = 0.1 x1 = np.random.normal(-5,2,int(n*dist_frac)) x2 = np.random.normal(5,3,int(n*(1-dist_frac))) x = np.concatenate((x1,x2)) np.random.shuffle(x) eval_points = np.linspace(np.min(x), np.max(x)) kde_sk = KernelDensity(bandwidth=1.0, kernel='gaussian') kde_sk.fit(x.reshape([-1,1])) y_sk = np.exp(kde_sk.score_samples(eval_points.reshape(-1,1))) kde_sp = gaussian_kde(x, bw_method=1.0) y_sp = kde_sp.pdf(eval_points) sns.kdeplot(x) plt.plot(eval_points, y_sk) plt.plot(eval_points, y_sp) plt.legend(['seaborn','scikit','scipy'])   If I change the scipy bandwith to 0.25, the result of both methods look approximately the same. 

Answer: What is meant by bandwidth in scipy.stats.gaussian_kde and sklearn.neighbors.KernelDensity is not the same. Scipy.stats.gaussian_kde uses a bandwidth factor https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html. For a 1-D kernel density estimation the following formula is applied: the bandwidth of sklearn.neighbors.KernelDensity = bandwidth factor of the scipy.stats.gaussian_kde * standard deviation of the sample For your estimation this probably means that your standard deviation equals 4. I would like to refer to Getting bandwidth used by SciPy's gaussian_kde function for more information.
Question: I'm trying to log into telegram using telethon with a number with two-step verification. I use this code, client = TelegramClient(f'sessions/1', API_ID, API_HASH) client.connect() phone = input('phone ; ') y = client.send_code_request(phone) x = client.sign_in(phone=phone, password=input('password : '), code=input('code :'))  But It still says that the account is two-step protected. Is there any easier way to do this without this method or... how can I properly use this method? I want to log into the account fully from the code without typing anything in the terminal (Here I used inputs just for testing. I will connect a GUI later where users can enter the details) so I don't think client.start() will work. and I'm a little confused when it comes to passing the parameters to client.start() method. Any help would be really appreciated. Thank you.

Answer: You also need to pass the phone_code_hash returned from client.send_code_request(phone). You could try (see the function call of sign_in with phone_code_hash and send_code_request): y = client.send_code_request(phone) client.sign_in(phone=phone, password=input('password : '), code=input('code :'), phone_code_hash=y.phone_code_hash)
Question: I am running this code below and it returned an error AttributeError: module 'keras.optimizers' has no attribute 'RMSprop'. I download tensorflow using pip install tensorflow. from keras import layers from keras import models model = models.Sequential() model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3))) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(128, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(128, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Flatten()) model.add(layers.Dense(512, activation='relu')) model.add(layers.Dense(1, activation='sigmoid')) model.summary() from keras import optimizers model.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop(lr=1e-4), metrics=['acc'])  Could anyone please help explain to me what's wrong with this? Thank you for your time.

Answer: As you said, you installed tensorflow (which includes keras) via pip install tensorflow, and not keras directly. Installing keras via pip install keras is not recommended anymore (see also the instructions here). This means that keras is available through tensorflow.keras. Instead of importing via from keras import optimizers, you should use from tensorflow.keras import optimizers.
Question: Everywhere i see that uWSGI and Gunicorn are recommended for production mode from everyone. However, there is a lot more suffering to operate with it, the python manage.py runserver is more faster, simpler, and the logging is also more visible if something goes wrong. Still, why not recommend the "python manage.py runserver" command for live production?

Answer: The runserver management command is optimized for different things from a web-server. Here are some things it does that are great for local development but would add unnecessary overhead in a production environment (source):  The development server automatically reloads Python code for each request, as needed When you start the server, and each time you change Python code while the server is running, the system check framework will check your entire Django project for some common errors Serves static files if the staticfiles contrib app is enabled (in a manner the docs describe as "grossly inefficient and probably insecure")  Meanwhile, production web-servers are designed to handle massively parallel workloads and are also under much higher security standards as they are the entry-point for all port 80/443 traffic to the server
Question: I have a dataframe df wit lots of coumnns and I would love to add another column with the name date that contains the date of today like in Excel the TODAY function. How can I do this?

Answer: Assuming your dataframe is named df, you can use the datetime library: from datetime import datetime df['new_column']= datetime.today().strftime('%Y-%m-%d')  @Henry Ecker raised the point that the same is possible in native pandas using pandas.Timestamp.today df['new_column'] = pd.Timestamp.today().strftime('%Y-%m-%d')
Question: I want to create a new boolean column in my dataframe that derives its value from the evaluation of two conditional statements on other columns in the same dataframe: columns = ["id", "color_one", "color_two"] data = spark.createDataFrame([(1, "blue", "red"), (2, "red", None)]).toDF(*columns) data = data.withColumn('is_red', data.color_one.contains("red") | data.color_two.contains("red"))  This works fine unless either color_one or color_two is NULL in a row. In cases like these, is_red is also set to NULL for that row instead of true or false: +-------+----------+------------+-------+ |id |color_one |color_two |is_red | +-------+----------+------------+-------+ | 1| blue| red| true| | 2| red| NULL| NULL| +-------+----------+------------+-------+  This means that PySpark is evaluating all of the clauses of the conditional statement rather than exiting early (via short-circuit evaluation) if the first condition happens to be true (like in row 2 of my example above). Does PySpark support the short-circuit evaluation of conditional statements? In the meantime, here is a workaround I have come up with to null-check each column: from pyspark.sql import functions as F color_one_is_null = data.color_one.isNull() color_two_is_null = data.color_two.isNull() data = data.withColumn('is_red', F.when(color_two_is_null, data.color_one.contains("red")) .otherwise(F.when(color_one_is_null, data.color_two.contains("red")) .otherwise(F.when(color_one_is_null & color_two_is_null, F.lit(False)) .otherwise(data.color_one.contains("red") | data.color_two.contains("red")))) ) 

Answer: I don't think Spark support short-circuit evaluation on conditionals as stated here https://docs.databricks.com/spark/latest/spark-sql/udf-python.html#:~:text=Spark%20SQL%20(including,short-circuiting%E2%80%9D%20semantics.:  Spark SQL (including SQL and the DataFrame and Dataset API) does not guarantee the order of evaluation of subexpressions. In particular, the inputs of an operator or function are not necessarily evaluated left-to-right or in any other fixed order. For example, logical AND and OR expressions do not have left-to-right “short-circuiting” semantics.  Another alternative way would be creating an array of column_one and column_two, then evaluate if the array contains 'red' using SQL EXISTS data = data.withColumn('is_red', F.expr("EXISTS(array(color_one, color_two), x -> x = 'red')")) data.show() +---+---------+---------+------+ | id|color_one|color_two|is_red| +---+---------+---------+------+ | 1| blue| red| true| | 2| red| null| true| | 3| null| green| false| | 4| yellow| null| false| | 5| null| red| true| | 6| null| null| false| +---+---------+---------+------+