eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here's two groupby operations on a pandas.DataFrame: import pandas d = pandas.DataFrame({"a": [1, 2, 3, 4, 5, 6], "b": [1, 2, 4, 3, -1, 5]}) grp1 = pandas.Series([1, 1, 1, 1, 1, 1]) ans1 = d.groupby(grp1).apply(lambda x: x.a * x.b.iloc[0]) grp2 = pandas.Series([1, 1, 1, 2, 2, 2]) ans2 = d.groupby(grp2).apply(lambda x: x.a * x.b.iloc[0]) print(ans1.reset_index(drop=True)) # a 0 1 2 3 4 5 # 0 1 2 3 4 5 6 print(ans2.reset_index(drop=True)) # 0 1 # 1 2 # 2 3 # 3 12 # 4 15 # 5 18 # Name: a, dtype: int64  I want the output in the format of ans2. If the grouping Series has more than one group (as in grp2), then there is no issue with the output format. However, when grouping Series has only one group (as in grp1), the output is a DataFrame with a single row. Why is this? How can I ensure that the output will always be like ans2 regardless of the number of groups in the grouping Series? Is there a quicker/better approach than  Checking if the output is a DataFrame and coercing into a Series Checking if the grouping Series has only one group and avoiding groupby if that's the case 

Answer: A simple solution is to return a DataFrame from apply: import pandas d = pandas.DataFrame({"a": [1, 2, 3, 4, 5, 6], "b": [1, 2, 4, 3, -1, 5]}) grp1 = pandas.Series([1, 1, 1, 1, 1, 1]) ans1 = d.groupby(grp1).apply(lambda x: x[['a']] * x.b.iloc[0]) grp2 = pandas.Series([1, 1, 1, 2, 2, 2]) ans2 = d.groupby(grp2).apply(lambda x: x[['a']] * x.b.iloc[0]) print(ans1.reset_index(drop=True)) # a # 0 1 # 1 2 # 2 3 # 3 4 # 4 5 # 5 6 print(ans2.reset_index(drop=True)) # a # 0 1 # 1 2 # 2 3 # 3 12 # 4 15 # 5 18  To understand why, the documentation of apply function is helpful. When the function given to apply returns a Series they are converted to a row and final output is a DataFrame with one row per group. So the behaviour of grp1 is actually expected. This begs the question why does the second case using grp2 return a Series. I think that is because the two groups return Series with different index values. Thus the results of the two groups are appended in a single row with multi-level indexing (as seen below). d = pandas.DataFrame({"a": [1, 2, 3, 4, 5, 6], "b": [1, 2, 4, 3, -1, 5]}) grp2 = pandas.Series([1, 1, 1, 2, 2, 2]) def func(x): z= x.a * x.b.iloc[0] print(z.index) return z ans2 = d.groupby(grp2).apply(func) # Int64Index([0, 1, 2], dtype='int64') # Int64Index([3, 4, 5], dtype='int64') print(ans2) # 1 0 1 # 1 2 # 2 3 # 2 3 12 # 4 15 # 5 18 # Name: a, dtype: int64
Question: I am new to Jupyter notebook and trying to see the some help about the functions. For example, when I print the help of statsmodels.OLS I got the following plain black and white help. Are there any python modules that colorize/beautify the help outputs? For example:  hightlight parameters names highlight the code example in python syntax hightlight and so on.  If there are not some modules, what would be the starting point, to colorize the parameters and the python codes? The example output of help is given below: 

Answer: You can try to beautify the help using rich library (in jupyter, you can install it, using the command !pip install rich). In particular, you could study the inspect method. For example, with the following code: from rich import inspect inspect(sm.OLS, help=True)  I get this output:
Question: I have a GTK3 GUI called by a simple Python 3 code. Icon is located in the /usr/share/icons/hicolor/scalable/actions/ directory. My current theme color is dark and icons look white. When I switch to white system theme GUI icons turn into black. But in my code icon looks as black instead of white when dark theme is activated. It works when I choose the icon name (icon-symbolic) from Glade program and save the UI file. Icon file is a simple black square .svg file (drawn in Inkscape). What is the solution for that? OS: Debian-like Linux, Python 3, GTK 3.24  Simple Python code: import gi gi.require_version('Gtk', '3.0') from gi.repository import Gtk, GdkPixbuf builder = Gtk.Builder() builder.add_from_file('test.ui') window1 = builder.get_object('window1') button1 = builder.get_object('button1') class Signals: def on_window1_destroy(self, widget): Gtk.main_quit() builder.connect_signals(Signals()) window1.set_icon_name("icon-symbolic") window1.show_all() Gtk.main()  Simple UI file: <?xml version="1.0" encoding="UTF-8"?> <!-- Generated with glade 3.38.2 --> <interface> <requires lib="gtk+" version="3.20"/> <object class="GtkWindow" id="window1"> <property name="can-focus">False</property> <property name="default-width">300</property> <property name="default-height">300</property> <child> <!-- n-columns=1 n-rows=1 --> <object class="GtkGrid"> <property name="visible">True</property> <property name="can-focus">False</property> <child> <object class="GtkButton" id="button1"> <property name="label" translatable="yes">Button 1</property> <property name="visible">True</property> <property name="can-focus">True</property> <property name="receives-default">True</property> </object> <packing> <property name="left-attach">0</property> <property name="top-attach">0</property> </packing> </child> </object> </child> </object> </interface> 

Answer: I have found a solution for automatically changing icon color on the window title bar. I have used Gtk.HeaderBar instead of default window title bar and added an Gtk.Image (its name is image_headerbar) to the left of the headerbar. Finally I have set image icon by using the following code and it worked: image_headerbar.set_from_icon_name("icon-symbolic", -1) Icon color changes to dark/white automatically when system theme changes to white/dark. I have tried several methods for dynamically changing icon color on the window title bar. But none of them worked if Gtk.HeaderBar is not used. But window title bar height is a bit bigger than default window title bars when Gtk.HeaderBar is used (tested on XFCE desktop environment).
Question: I'm struggling with a numba error Untyped global name 'is_a_subset': Cannot determine Numba type of <class 'numba.np.ufunc.gufunc.GUFunc'> This usually means I have fumbled and used a method that isn't supported by numba. The following code fails. @guvectorize("(n),(n)->(n)",nopython=True) def is_a_subset(x,y,out): out[:]=np.array([item in x for item in y]) @njit() def test(x,y,z): is_a_subset(x,y,z) return z.mean() x=np.array([[1,2,3],[3,2,1]]) y=np.array([[3,6,1],[1,2,3]]) z = np.empty_like(x) test(x,y,z)  However removing njit on the test function makes everything work. def test(x,y,z): is_a_subset(x,y,z) return z.mean()  Why is numba struggling to resolve types when in no-python mode? I had also tried without different results @guvectorize(["f8[:],f8[:],f8[:]"],"(n),(n)->(n)",nopython=True) def is_a_subset(x,y,out): out[:]=np.array([item in x for item in y]) 

Answer: I am using Numba 0.53.1 and can replicate this error. This blog on the dynamic dispatch update to guvectorize in Numba 0.53 mentions this at the end (emphasis added):  In the future we would like to bring the @guvectorize capabilities closer to the @vectorize ones. For instance, currently it is not possible to call a guvectorize function from a jitted (@jit) function.  There is a similar open issue with vectorize, but it demonstrates that @vectorize functions can be called in @jit functions, just that it's restricted to the default target = "cpu".
Question: Two DataFrames have city names that are not formatted the same way. I'd like to do a Left-outer join and pull geo field for all partial string matches between the field City in both DataFrames. import pandas as pd df1 = pd.DataFrame({ 'City': ['San Francisco, CA','Oakland, CA'], 'Val': [1,2] }) df2 = pd.DataFrame({ 'City': ['San Francisco-Oakland, CA','Salinas, CA'], 'Geo': ['geo1','geo2'] })  Expected DataFrame upon join:  City Val Geo San Francisco, CA 1 geo1 Oakland, CA 2 geo1 

Answer: Update: the fuzzywuzzy project has been renamed to thefuzz and moved here You can use thefuzz package and the function extractOne: # Python env: pip install thefuzz # Anaconda env: pip install thefuzz # -> thefuzz is not yet available on Anaconda (2021-09-18) # -> you can use the old package: conda install -c conda-forge fuzzywuzzy from thefuzz import process best_city = lambda x: process.extractOne(x, df2["City"])[2] # See note below df1['Geo'] = df2.loc[df1["City"].map(best_city).values, 'Geo'].values  Output: >>> df1 City Val Geo 0 San Francisco, CA 1 geo1 1 Oakland, CA 2 geo1  Note: extractOne return a tuple of 3 values from the best match: the City name from df2 [0], the accuracy score [1] and the index [2] (<- the one I use).
Question: I recently installed Cron via jupyterlab_scheduler in the anaconda extensions in a conda environment I usually work in. This was to schedule my jupyterlab notebooks. However, there was a problem with the application and so I deleted it. Though it seems to have left some of its features like turning the cell orange and leaving an asterisk to the left of the cell number. The picture below demonstrates this:  I created a new environment though it seems to still be affecting it other environs. Is there any reason why this is still happening? Its a problem because previously, when I use to undo (ctrl + z), it used to undo everything in the cell and only the cell in question. But now it undoes everything across all cells. This is a problem for me as it changes the overall code I am working with. Any idea how to rectify this?

Answer: As explained in the JupyterLab 3.1 changelog, specifically the user-facing changes section, a new new visual indicator was introduced to highlight cells in which the code changed in the editor since last execution:  The indicator is currently implemented by changing the cell collapser and the cell execution counter color to orange, and adding a filled orange circle icon left execution counter. Hopefully, this will improve the situational awareness of the users and lead to more consistent state of the notebooks on save. If you come to like this solution you may be interested in using nbsafety which takes it a step further by actually analysing the dependencies and preventing out-of-order execution.
Question: I am writing a small code to calculate the fourth derivative using the method of finite differences in tensorflow. This is as follows: def action(y,x): #spacing between points. h = (x[-1] - x[0]) / (int(x.shape[0]) - 1) #fourth derivative dy4 = (y[4:] - 4*y[3:-1] + 6*y[2:-2] - 4*y[1:-3] + y[:-4])/(h*h*h*h) return dy4 x = tf.linspace(0.0, 30, 1000) y = tf.tanh(x) dy4 = action(y,x) sess = tf.compat.v1.Session() plt.plot(sess.run(dy4))  This results in the following graph:  However if I use essentially the same code but just using numpy, the results are much cleaner: def fourth_deriv(y, x): h = (x[-1] - x[0]) / (int(x.shape[0]) - 1) dy = (y[4:] - 4*y[3:-1] + 6*y[2:-2] - 4*y[1:-3] + y[:-4])/(h*h*h*h) return dy x = np.linspace(0.0, 30, 1000) test = fourth_deriv(np.tanh(x), x) plt.plot(test)  Which gives:  What is the issue here? I was thinking at first that the separation between points could be too small to give an accurate computation, but clearly, that's not the case if numpy can handle it fine.

Answer: The issue is related to the choice of floating-point types.  tf.linspace automatically selects tf.float32 as its type, while np.linspace creates a float64 array, which has much more precision.  Making the following modification: start = tf.constant(0.0, dtype = tf.float64) end = tf.constant(30.0, dtype = tf.float64) x = tf.linspace(start, end, 1000)  causes a smooth plot to appear:  It's worth noting further that Tensorflow does include an automatic differentiation, which is crucial for machine learning training and is hence well-tested - you can use gradient tapes to access it and evaluate a fourth derivative without the imprecision of numeric differentiation using finite differences: with tf.compat.v1.Session() as sess2: x = tf.Variable(tf.linspace(0, 30, 1000)) sess2.run(tf.compat.v1.initialize_all_variables()) with tf.GradientTape() as t4: with tf.GradientTape() as t3: with tf.GradientTape() as t2: with tf.GradientTape() as t1: y = tf.tanh(x) der1 = t1.gradient(y, x) der2 = t2.gradient(der1, x) der3 = t3.gradient(der2, x) der4 = t4.gradient(der3, x) print(der4) plt.plot(sess2.run(der4))  The accuracy of this method is far better than can be achieved using finite difference methods. The following code compares the accuracy of auto diff with the accuracy of the finite difference method: x = np.linspace(0.0, 30, 1000) sech = 1/np.cosh(x) theoretical = 16*np.tanh(x) * np.power(sech, 4) - 8*np.power(np.tanh(x), 3)*np.power(sech,2) finite_diff_err = theoretical[2:-2] - from_finite_diff autodiff_err = theoretical[2:-2] - from_autodiff[2:-2] print('Max err with autodiff: %s' % np.max(np.abs(autodiff_err))) print('Max err with finite difference: %s' % np.max(np.abs(finite_diff_err))) line, = plt.plot(np.log10(np.abs(autodiff_err))) line.set_label('Autodiff log error') line2, = plt.plot(np.log10(np.abs(finite_diff_err))) line2.set_label('Finite difference log error') plt.legend()  and yields the following output: Max err with autodiff: 3.1086244689504383e-15 Max err with a finite difference: 0.007830900165363808  and the following plot (the two lines overlap after around 600 on the X-axis):
Question: I am trying to work with Featuretools to develop an automated feature engineering workflow for the customer churn dataset. The end outcome is a function that takes in a dataset and label times for customers and builds a feature matrix that can be used to train a machine learning model. As part of this exercise I am trying to execute the below code for plotting a histogram and got "TypeError: import_optional_dependency() got an unexpected keyword argument 'errors' ". Please help resolve this TypeError. import matplotlib.pyplot as plt %matplotlib inline plt.style.use('fivethirtyeight') plt.rcParams['figure.figsize'] = (10, 6) trans.loc[trans['actual_amount_paid'] < 250, 'actual_amount_paid'].dropna().plot.hist(bins = 30) plt.title('Distribution of Actual Amount Paid')  Below is the full error I received:  --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-32-7e19affd5fc1> in <module> 4 plt.rcParams['figure.figsize'] = (10, 6) 5 ----> 6 trans.loc[trans['actual_amount_paid'] < 250, 'actual_amount_paid'].dropna().plot.hist(bins = 30) 7 plt.title('Distribution of Actual Amount Paid') ~\anaconda3\lib\site-packages\pandas\core\ops\common.py in new_method(self, other) 63 break 64 if isinstance(other, cls): ---> 65 return NotImplemented 66 67 other = item_from_zerodim(other) ~\anaconda3\lib\site-packages\pandas\core\arraylike.py in __lt__(self, other) 35 def __ne__(self, other): 36 return self._cmp_method(other, operator.ne) ---> 37 38 @unpack_zerodim_and_defer("__lt__") 39 def __lt__(self, other): ~\anaconda3\lib\site-packages\pandas\core\series.py in _cmp_method(self, other, op) 4937 -------- 4938 >>> s = pd.Series(range(3)) -> 4939 >>> s.memory_usage() 4940 152 4941 ~\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py in comparison_op(left, right, op) 248 lvalues = ensure_wrapped_if_datetimelike(left) 249 rvalues = ensure_wrapped_if_datetimelike(right) --> 250 251 rvalues = lib.item_from_zerodim(rvalues) 252 if isinstance(rvalues, list): ~\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py in _na_arithmetic_op(left, right, op, is_cmp) 137 138 def _na_arithmetic_op(left, right, op, is_cmp: bool = False): --> 139 140 Return the result of evaluating op on the passed in values. 141 ~\anaconda3\lib\site-packages\pandas\core\computation\expressions.py in <module> 17 from pandas._typing import FuncType 18 ---> 19 from pandas.core.computation.check import NUMEXPR_INSTALLED 20 from pandas.core.ops import roperator 21 ~\anaconda3\lib\site-packages\pandas\core\computation\check.py in <module> 1 from pandas.compat._optional import import_optional_dependency 2 ----> 3 ne = import_optional_dependency("numexpr", errors="warn") 4 NUMEXPR_INSTALLED = ne is not None 5 if NUMEXPR_INSTALLED: TypeError: import_optional_dependency() got an unexpected keyword argument 'errors' 

Answer: Try to upgrade pandas: pip install pandas --upgrade
Question: I was trying to send http/https requests via proxy (socks5), but I can't understand if the problem is in my code or in the proxy. I tried using this code and it gives me an error: requests.exceptions.ConnectionError: SOCKSHTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000001B656AC9608>: Failed to establish a new connection: Connection closed unexpectedly'))  This is my code: import requests url = "https://www.google.com" proxies = { "http":"socks5://fsagsa:sacesf241_country-darwedafs_session-421dsafsa@x.xxx.xxx.xx:31112", "https":"socks5://fsagsa:sacesf241_country-darwedafs_session-421dsafsa@x.xxx.xxx.xx:31112", } headers = { "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "Sec-Gpc": "1", "Sec-Fetch-Site": "same-origin", "Sec-Fetch-User": "?1", "Accept-Encoding": "gzip, deflate, br", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Dest": "document", "Accept-Language": "en-GB,en;q=0.9" } r = requests.get(url, headers = headers, proxies = proxies) print(r)  Then, I checked the proxy with an online tool The tool manages to send requests through the proxy. . So the problem is in this code? I can't figure out what's wrong. Edit (15/09/2021) I added headers but the problem is still there.

Answer: Create a local server/mock to handle the request using pytest or some other testing framework with responses library to eliminate variables external to your application/script. I’m quite sure Google will reject requests with empty headers. Also, ensure you installed the correct dependencies to enable SOCKS proxy support in requests (python -m pip install requests[socks]). Furthermore, if you are making a remote request to connect to your proxy you must change socks5 to socks5h in your proxies dictionary. References pytest: https://docs.pytest.org/en/6.2.x/ responses: https://github.com/getsentry/responses requests[socks]: https://docs.python-requests.org/en/master/user/advanced/#socks
Question: Say I have a matrix Y of random float numbers from 0 to 10 with shape (10, 3): import numpy as np np.random.seed(99) Y = np.random.uniform(0, 10, (10, 3)) print(Y)  Output: [[6.72278559 4.88078399 8.25495174] [0.31446388 8.08049963 5.6561742 ] [2.97622499 0.46695721 9.90627399] [0.06825733 7.69793028 7.46767101] [3.77438936 4.94147452 9.28948392] [3.95454044 9.73956297 5.24414715] [0.93613093 8.13308413 2.11686786] [5.54345785 2.92269116 8.1614236 ] [8.28042566 2.21577372 6.44834702] [0.95181622 4.11663239 0.96865261]]  I am now given a matrix X with same shape that can be seen as obtained by adding small noises to Y and then shuffling the rows: X = np.random.normal(Y, scale=0.1) np.random.shuffle(X) print(X)  Output: [[ 4.04067271 9.90959141 5.19126867] [ 5.59873104 2.84109306 8.11175891] [ 0.10743952 7.74620162 7.51100441] [ 3.60396019 4.91708372 9.07551354] [ 0.9400948 4.15448712 1.04187208] [ 2.91884302 0.47222752 10.12700505] [ 0.30995155 8.09263241 5.74876947] [ 1.11247872 8.02092335 1.99767444] [ 6.68543696 4.8345869 8.17330513] [ 8.38904822 2.11830619 6.42013343]]  Now I want to sort the matrix X based on Y by row. I already know each pair of column values in each matching pair of rows are not different from each other more than a tolerance of 0.5. I managed to write the following code and it is working fine. def sort_X_by_Y(X, Y, tol): idxs = [next(i for i in range(len(X)) if all(abs(X[i] - row) <= tol)) for row in Y] return X[idxs] print(sort_X_by_Y(X, Y, tol=0.5))  Output: [[ 6.68543696 4.8345869 8.17330513] [ 0.30995155 8.09263241 5.74876947] [ 2.91884302 0.47222752 10.12700505] [ 0.10743952 7.74620162 7.51100441] [ 3.60396019 4.91708372 9.07551354] [ 4.04067271 9.90959141 5.19126867] [ 1.11247872 8.02092335 1.99767444] [ 5.59873104 2.84109306 8.11175891] [ 8.38904822 2.11830619 6.42013343] [ 0.9400948 4.15448712 1.04187208]]  However, in reality I am sorting (1000, 3) matrices and my code is way too slow. I feel like there should be more numpyic way to code this. Any suggestions?

Answer: This is a vectorized version of your algorithm. It runs ~26.5x faster than your implementation for 1000 samples. But an additional boolean array with shape (1000,1000,3) is created. There is a chance that rows will have similar values within the tolerance and a wrong row is selected. tol = .5 X[(np.abs(Y[:, np.newaxis] - X) <= tol).all(2).argmax(1)]  Output array([[ 6.68543696, 4.8345869 , 8.17330513], [ 0.30995155, 8.09263241, 5.74876947], [ 2.91884302, 0.47222752, 10.12700505], [ 0.10743952, 7.74620162, 7.51100441], [ 3.60396019, 4.91708372, 9.07551354], [ 4.04067271, 9.90959141, 5.19126867], [ 1.11247872, 8.02092335, 1.99767444], [ 5.59873104, 2.84109306, 8.11175891], [ 8.38904822, 2.11830619, 6.42013343], [ 0.9400948 , 4.15448712, 1.04187208]])  More robust solutions with L1-norm X[np.abs(Y[:, np.newaxis] - X).sum(2).argmin(1)]  Or L2-norm X[((Y[:, np.newaxis] - X)**2).sum(2).argmin(1)]
Question: So, I found the following example in bcrypt docs: password = b"super secret password" hashed = bcrypt.hashpw(password, bcrypt.gensalt()) if bcrypt.checkpw(password, hashed): print("It Matches!") else: print("It Does not Match :(")  And it seems to work. But I don't understand how. Shouldn't we use salt to generate a hash for checking? I mean, we generated salt once and didn't save it in a variable. But then we want to compare the hash and the password with the function checkpw, but how does it know which salt to use to generate a hash for comparison?

Answer: The generated "hash" also contains the salt. It is in the Modular Crypt Format, documented here (thanks @Masklinn) $2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy |<--- salt --->||<---- confirmation hash ---->|  The "2a" part gives information on the modular hash being used, "10" is the logarithmic cost parameter (i.e. the algorithm is to be iterated 210 times). So, to verify that a password matches, you'll restart the bcrypt using the decoding of N9qo8uLOickgx2ZMRZoMye as a salt.