eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Consider the following code: import pandas import numpy strs = ['custom','sort']*5 df = pandas.DataFrame( { 'string': strs, 'number': numpy.random.randn(len(strs)), } ) sort_string_like_this = {'sort': 0, 'custom': 1} print(df.sort_values(['string','number'], key=lambda x: x.map(sort_string_like_this)))  which prints  string number 1 sort -0.074041 3 sort 1.057676 5 sort -0.612289 7 sort 0.757922 9 sort 0.671288 0 custom -0.339373 2 custom -0.320231 4 custom -1.125380 6 custom 2.120829 8 custom -0.031580  I would like to sort it according to the column string using a custom ordering as given by the dictionary and the column number using the natural ordering of numbers. How can this be done?

Answer: You can use a condition in the sort key function: df.sort_values( ["string", "number"], key=lambda x: x.map(sort_string_like_this) if x.name == "string" else x, )   string number 7 sort -1.673626 3 sort -0.212634 5 sort -0.071417 9 sort 0.413497 1 sort 0.489508 8 custom -1.787110 0 custom 0.230875 4 custom 0.535791 2 custom 0.671282 6 custom 2.119993
Question: I have written following code to rotate a curve by specified angle and return new equation. I know when we want to rotate the axes by an angle A then new coords become X=xcosA-ysinA and Y=xsinA+ycosA. when i test my fxn on the hyperbola x^2+y^2=1 . Expected equations is 2xy-1=0 but my fxn gives -2xy-1=0 Where I am doing wrong? import sympy as sp @display_mathjax_and_replace def rotate(f,theta): x, y, a, b = sp.symbols('x y a b') rotation_matrix = sp.Matrix([[sp.cos(theta),-sp.sin(theta)],[sp.sin(theta), sp.cos(theta)]]) transformed_coords = rotation_matrix * sp.Matrix([a, b]) g = f.subs({x: transformed_coords[0], y: transformed_coords[1]}) return g  I have created following decorator to present it beautifully: def display_mathjax_and_replace(func): def wrapper(*args, **kwargs): result = func(*args, **kwargs) result = result.subs({'a': 'x', 'b': 'y'}) simplified_expr = sp.simplify(result) display(Math(sp.latex(sp.Eq(simplified_expr, 0)))) return wrapper  I call my code : x,y = sp.symbols('x y') f = x**2 - y**2 -1 rotate(f,sp.pi/4)  Output: -2xy-1 Expected output:2xy-1

Answer: You swapped the sign on the sin terms of the transformation matrix. Here is how it should look like for a positive rotation: def rotate_1(f, theta): x, y, a, b = sp.symbols('x y a b') rotation_matrix = sp.Matrix([[sp.cos(theta),sp.sin(theta)],[-sp.sin(theta), sp.cos(theta)]]) transformed_coords = rotation_matrix * sp.Matrix([a, b]) g = f.subs({x: transformed_coords[0], y: transformed_coords[1]}) return g rotate(f, sp.pi/4).simplify() # out: 2*a*b - 1  If you use SymPy Plotting Backend and an interactive environment like Jupyter Notebook, you can perform nice visualization that helps you catch any eventual mistake. For example: %matplotlib widget from sympy import * from spb import * def symbolic_rotate(f, theta, x, y): a, b = var("a, b") rotation_matrix = Matrix([[cos(theta),sin(theta)],[-sin(theta), cos(theta)]]) transformed_coords = rotation_matrix * Matrix([a, b]) g = f.subs({x: transformed_coords[0], y: transformed_coords[1]}) g = g.subs({a: x, b: y}) return g var("x, y, theta, a, b") f = x**2 - y**2 -1 graphics( implicit_2d(f, (x, -10, 10), (y, -10, 10), label="f"), implicit_2d(symbolic_rotate(f, theta, x, y), (x, -10, 10), (y, -10, 10), label="f rot", params={theta: (pi/6, 0, 2*pi)}), aspect="equal" )   symbolic_rotate creates a new expression based on a symbolic angle of rotation. Then, we can plot the original hyperbole and the rotated one and play with a slider. Here, you can visually verify if the transformation matrix is correct.
Question: I want to generate a twodimensional array in Python and I would like to iterate through each element and take an average. An element i should be averaged using the 8 surrounding array elements (including element i). I generated the twodimensional array with a frame of zeros using Forming a frame of zeros around a matrix in python. A = np.array([[1,2,3],[4,5,6],[7,8,9]]) x,y = A.shape n = 1 B = np.zeros((x+2*n,y+2*n),dtype=int) B[n:x+n, n:y+n] = A print(B)  What is the easiest way to take the average?

Answer: What you want is a 2D convolution, use scipy.signal.convolve2d with numpy.ones to get the sum: from scipy.signal import convolve2d out = convolve2d(B, np.ones((3, 3), dtype='int'), mode='same')  Output: array([[ 1, 3, 6, 5, 3], [ 5, 12, 21, 16, 9], [12, 27, 45, 33, 18], [11, 24, 39, 28, 15], [ 7, 15, 24, 17, 9]])  If you want the small matrix, no need to pad with zeros: convolve2d(A, np.ones((3, 3), dtype='int'), mode='same') # array([[12, 21, 16], # [27, 45, 33], # [24, 39, 28]])  For the average, repeat the same operation with an array od ones and divide: kernel = np.ones((3, 3), dtype='int') out = (convolve2d(A, kernel, mode='same') /convolve2d(np.ones_like(A), kernel, mode='same') )  output: array([[3. , 3.5, 4. ], [4.5, 5. , 5.5], [6. , 6.5, 7. ]])
Question: An array of integers is given. It is necessary to find the maximum product of the distance between a pair of elements and the minimum element of this pair. For example, [2, 5, 2, 2, 1, 5, 2] -> 20, 5 and 5 (5 * (5-1)); [1, 2] -> 1 In the first example, the maximum product is obtained if we take 5 and 5(a[1] and a[5]). The minimum element of the pair is 5. The distance between the elements: 5 - 1 = 4. Result: 5 * 4 = 20 In the second example, there is only one pair. The minimum element is 1. The distance between the elements is 1. Result: 1*1 = 1 not an effective solution: a = [2,5,2,2,1,5,2] res = -10**9 for i in range(len(a)-1): for j in range(i+1,len(a)): res = max(res, min(a[i],a[j])*(j-i)) print(res)  The code is slow to work. How can I make it more efficient?

Answer: O(n log n): from itertools import accumulate from bisect import bisect_left a = [2,5,2,2,1,5,2] print(max( x * (j - bisect_left(m, x)) for a in [a, a[::-1]] for m in [list(accumulate(a, max))] for j, x in enumerate(a) ))  Attempt This Online! Assume the best pair has the smaller-or-equal value as the right value. For each right value, find the furthest left value larger or equal (so that the right value is indeed smaller or equal) and evaluate that pair. Do the whole thing both for the given array and for it reversed, which covers the case that the best pair has the smaller-or-equal value as the left value. Inspired by Nijat Mursali's opening paragraph.
Question: I have two pl.DataFrames: from datetime import date import polars as pl df1 = pl.DataFrame( { "symbol": [ "sec1", "sec1", "sec1", "sec1", "sec1", "sec1", "sec2", "sec2", "sec2", "sec2", "sec2", ], "date": [ date(2021, 9, 14), date(2021, 9, 15), date(2021, 9, 16), date(2021, 9, 17), date(2021, 8, 31), date(2020, 12, 31), date(2021, 9, 14), date(2021, 9, 15), date(2021, 8, 31), date(2021, 12, 30), date(2020, 12, 31), ], "price": range(11), } ) df2 = pl.DataFrame( { "symbol": ["sec1", "sec2"], "current_date": [date(2021, 9, 17), date(2021, 9, 15)], "mtd": [date(2021, 8, 31), date(2021, 8, 31)], "ytd": [date(2020, 12, 31), date(2020, 12, 30)], } ) with pl.Config(tbl_rows=-1): print(df1) print(df2) shape: (11, 3) ┌────────┬────────────┬───────┐ │ symbol ┆ date ┆ price │ │ --- ┆ --- ┆ --- │ │ str ┆ date ┆ i64 │ ╞════════╪════════════╪═══════╡ │ sec1 ┆ 2021-09-14 ┆ 0 │ │ sec1 ┆ 2021-09-15 ┆ 1 │ │ sec1 ┆ 2021-09-16 ┆ 2 │ │ sec1 ┆ 2021-09-17 ┆ 3 │ │ sec1 ┆ 2021-08-31 ┆ 4 │ │ sec1 ┆ 2020-12-31 ┆ 5 │ │ sec2 ┆ 2021-09-14 ┆ 6 │ │ sec2 ┆ 2021-09-15 ┆ 7 │ │ sec2 ┆ 2021-08-31 ┆ 8 │ │ sec2 ┆ 2021-12-30 ┆ 9 │ │ sec2 ┆ 2020-12-31 ┆ 10 │ └────────┴────────────┴───────┘ shape: (2, 4) ┌────────┬──────────────┬────────────┬────────────┐ │ symbol ┆ current_date ┆ mtd ┆ ytd │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ date ┆ date ┆ date │ ╞════════╪══════════════╪════════════╪════════════╡ │ sec1 ┆ 2021-09-17 ┆ 2021-08-31 ┆ 2020-12-31 │ │ sec2 ┆ 2021-09-15 ┆ 2021-08-31 ┆ 2020-12-30 │ └────────┴──────────────┴────────────┴────────────┘  I need to filter the prices of df1 for each group with the respective dates from df2. I need to incorporate all columns of type date. The number of these columns in df2 might not be fixed. I am looking for the following result: shape: (11, 3) ┌────────┬────────────┬───────┐ │ symbol ┆ date ┆ price │ │ --- ┆ --- ┆ --- │ │ str ┆ date ┆ i64 │ ╞════════╪════════════╪═══════╡ │ sec1 ┆ 2021-09-17 ┆ 3 │ │ sec1 ┆ 2021-08-31 ┆ 4 │ │ sec1 ┆ 2020-12-31 ┆ 5 │ │ sec2 ┆ 2021-09-15 ┆ 7 │ │ sec2 ┆ 2021-08-31 ┆ 8 │ │ sec2 ┆ 2020-12-30 ┆ 9 │ └────────┴────────────┴───────┘  I was thinking of filtering df1 by symbol and then do a join operation for every individual date column of df2. I would then subsequently concatenate the resulting dataframes. However, there's probably a much more elegant solution.

Answer: You can unpivot, then join: df1.join( df2.unpivot(index='symbol', value_name='date').drop('variable'), on=['symbol', 'date'], how='inner', )  Output: ┌────────┬────────────┬───────┐ │ symbol ┆ date ┆ price │ │ --- ┆ --- ┆ --- │ │ str ┆ date ┆ i64 │ ╞════════╪════════════╪═══════╡ │ sec1 ┆ 2021-09-17 ┆ 3 │ │ sec1 ┆ 2021-08-31 ┆ 4 │ │ sec1 ┆ 2020-12-31 ┆ 5 │ │ sec2 ┆ 2021-09-15 ┆ 7 │ │ sec2 ┆ 2021-08-31 ┆ 8 │ └────────┴────────────┴───────┘
Question: I have a dataset i'm trying to remove duplicate entries from. The lazyframe i'm working with is structured like this: df = pl.from_repr(""" ┌──────┬────────────┬──────────────────┬───────┐ │ id ┆ title ┆ type ┆ type2 │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str ┆ i64 │ ╞══════╪════════════╪══════════════════╪═══════╡ │ 1001 ┆ Research A ┆ journal article ┆ 35 │ │ 1002 ┆ Research B ┆ book chapter ┆ 41 │ │ 1003 ┆ Research C ┆ journal article ┆ 35 │ │ 1004 ┆ Research D ┆ conference paper ┆ 42 │ │ 1001 ┆ Research E ┆ journal article ┆ 35 │ │ 1002 ┆ Research F ┆ journal article ┆ 41 │ │ 1003 ┆ Research G ┆ ┆ 41 │ │ 1002 ┆ Research I ┆ book chapter ┆ 41 │ │ 1003 ┆ Research J ┆ journal article ┆ 35 │ └──────┴────────────┴──────────────────┴───────┘ """)  I want to remove entries that have the same id, but there are actually different cases:  The duplicates have the same type (e.g. 1001): keep the first one. The duplicates have a different type: discard the ones with an empty string ("") as type, and then keep only the entries that respect the following pairs of type and type2:  dict_df = pl.DataFrame({ "type": ['journal article', 'book chapter', 'book chapter'], "type2": [35, 41, 42] })  Expected output    id[i64] title[str] type[str] type2[i64]     1001 Research A journal article 35   1002 Research B book chapter 41   1003 Research C journal article 35   1004 Research D conference paper 42     1001: same type, keep the first one 1002: different type, keep the first occurrence of the entries with pair {'book chapter': 41} 1003: different type, discard the entries with an empty type and keep the first occurrence 1004: not a duplicate  I've tried many things, mainly using the pl.when() expression, but I cannot work out a way to filter the groups. ( df .sort('type', descending=True) .group_by("id") .agg([ pl.when(pl.col("type").n_unique() > 1) .then( ... ) .otherwise(pl.all().first()) ]) ) 

Answer:  Create a preference table indicating how much you want each combination:  preference = pl.DataFrame({ "type": ["journal article", "book chapter", "book chapter"], "iris_type": [35, 41, 42], "preference": [0, 1, 2] })   Join the preference table with your data table:  joined = df.lazy().join(preference.lazy(), on=["type", "iris_type"], how="left")   Sort the joined table on the preference, pick the first one in each group and drop the preference column:  out = ( joined.sort("preference", descending=True, nulls_last=True) .group_by("id") .first() .drop("preference") .collect() )
Question: I'd like to make an abstraction of one of my api classes to resolve the following problem. Let's say I have a base class like: class AbstractAPI(ABC): @abstractmethod def create(self): pass @abstractmethod def delete(self): pass  And a concrete class: class API(AbstractAPI): def create(self): print("create") def delete(self): print("delete")  When requests come in, I don't have access to my API instance. Both due multithreading and to avoid some circular imports. At that point I do know which method I would like to call later. So my plan was to put a function pointer of AbstractAPI onto a queue and wait until I do have access to the API instance. function_pointer=AbstractAPI.create # later on ... function_pointer(ConcreteAPIInstance)  At that point call the function pointer onto the API instance, ba da bim, ba da boom. That does not work. Of course calling the function pointer of AbstractAPI onto an API instance calls the empty AbstractAPI method. Nothing happens. Is there a way to make this work?

Answer: Rather than directly referencing the abstract class method function_pointer=AbstractAPI.create, you can write a function that calls the named create() method on a given object. function_pointer = lambda api : api.create()  or def function_pointer(api): return api.create()
Question: I have a 2D numpy matrix: arr = np.array([(1, 2), (6, 0), (3, 3), (5, 4)])  I am trying to get the output: [5, 4] I have tried to do the following: max_arr = np.max(arr, axis=0)  But this finds the largest value in each column of the matrix, regardless of the vectors from which those values come from. Maybe there is some function that will consider the sum of each vector in the array? The idea is to use only the optimized NumPy functionality in the least steps possible.

Answer: Compute the sum per row, then get the position of the maximum with argmax and use this for indexing: out = arr[arr.sum(axis=1).argmax()]  Output: array([5, 4]) Intermediates: arr.sum(axis=1) # array([3, 6, 6, 9]) arr.sum(axis=1).argmax() # 3  variant: all maxes If you can have multiple maxima, you can use instead: out = arr[(s:=arr.sum(axis=1)) == s.max()]  Or for python < 3.8: s = arr.sum(axis=1) out = arr[s == s.max()]  Output: array([[5, 4]])  Note the 2D output, you would get multiple rows if several have the same maximum sum.
Question: Using Python lists: a = [[0, 1], [3, 4]] b = [0, 2] print(b in a)  I'm getting False as an output, but with Numpy arrays: a = np.array([[0, 1], [3, 4]]) b = np.array([0, 2]) print(b in a)  I'm getting True as an output. What is an equivalent of in operator above for 2D Numpy arrays?

Answer: In the NumPy array case, b in a is interpreted as checking if any element of b is in a, rather than checking for the presence of b as a whole array. You can use the numpy.all function along with numpy.any to compare rows: a = np.array([[0, 1], [3, 4]]) b = np.array([0, 2]) is_row_present = np.any(np.all(a == b, axis=1)) print(is_row_present) >>> False
Question: I have the string 'abç' which in UTF-8 is b'ab\xc3\xa7'. I want it in UTF-16, but not this way: b'ab\xc3\xa7'.decode('utf-8').encode('utf-16-be')  which gives me: b'\x00a\x00b\x00\xe7' The answer I want is the UTF-16 code units, that is, a list of int:  [32, 33, 327]  Is there any straightforward way to do that? And of course, the reverse. Given a list of ints which are UTF-16 code units, how do I convert that to UTF-8?

Answer: The simple solution that may work in many cases would be something like: def sort_of_get_utf16_code_units(s): return list(map(ord, s)) print(sort_of_get_utf16_code_units('abç')  Output: [97, 98, 231]  However, that doesn't work for characters outside the Basic Multilingual Plane (BMP): print(sort_of_get_utf16_code_units('😊'))  Output is the Unicode code point: [128522]  Where you might have expected the code units (as your question states): [55357, 56842]  To get that: def get_utf16_code_units(s): utf16_bytes = s.encode('utf-16-be') return [int.from_bytes(utf16_bytes[i:i+2]) for i in range(0, len(utf16_bytes), 2)] print(get_utf16_code_units('😊'))  Output: [55357, 56842]  Doing the reverse is similar: def utf16_code_units_to_string(code_units): utf16_bytes = b''.join([unit.to_bytes(2, byteorder='big') for unit in code_units]) return utf16_bytes.decode('utf-16-be') print(utf16_code_units_to_string([55357, 56842]))  Output: 😊  The byteorder is 'big' by default, but it doesn't hurt to be specific there.
Question: I'm trying to iterate through a list of datraframes and do two things, replace blanks with "_" (which I've done) and add a suffix to the first column of each dataframe. I know I can access the first column of the each dataframe via the print row within the loop below, but I'm having trouble adding the suffix to the first column. Can someone please help? Sample data: import pandas as pd px_data = {'Date': ['8/11/18', '8/12/18', '8/13/18', '8/14/18'], 'A df': [58.63, 21.25, 19.17, 18.8], 'B df': [35,105,27,98]} SC_data = {'Date': ['8/11/18', '8/12/18', '8/13/18', '8/14/18'], 'A df': [20.50, 6, 82, 74.6], 'B df': [74,62,8,99]} SMA_data = {'Date': ['8/11/18', '8/12/18', '8/13/18', '8/14/18'], 'A df': [2, 95.3, 39, 68.27], 'B df': [58,37,74,11]} px = pd.DataFrame(px_data) SC = pd.DataFrame(SC_data) SMA = pd.DataFrame(SMA_data) dfs=[px,SC,SMA] for i in range(len(dfs)): dfs[i].columns = dfs[i].columns.str.replace(' ', '_') print(dfs[i].columns[0]) 

Answer: After replacing blanks with underscore, you can add suffix to the first columns of each dataframe in the following way: first_column = dfs[i].columns[0] # get first column dfs[i].rename(columns={first_column: first_column + '_suffix'}, inplace=True) # Add a suffix to the first column
Question: I'm using a faceted chart in Altair in order to split a lengthy timeline into multiple rows. My initial dataset is a pandas dataframe with "Start" and "End" timestamp columns and a "Product" string column. I bin the dataset into roughly equal rows by evenly dividing the time range: timestamp_normalized = (data.Start - data.Start[0]) / (data.Start.iloc[-1] - data.Start[0]) # range from 0 to 1 data['Row'] = (timestamp_normalized * 3 - 1e-9).astype(int) # divide into 3 bins  Then I draw the facet chart like this: import altair as alt alt.Chart(data).mark_bar().encode( x=alt.X('Start', title='') x2='End', color=alt.Color('Product', scale=alt.Scale(scheme='dark2')) ).properties( width=800, height=50 ).facet( row=alt.Row('Row', title=''), ).resolve_scale( x='independent', )  This produces the right chart, but unfortunately the bin indices (which are completely irrelevant, as they only serve to split it into three pieces) are shown on the left side. Is there any way to hide these? 

Answer: To remove both the header title and labels you can use alt.Row('Row').header(None). You could also control them individually via alt.Row('Row').header(title=None, labels=False) For versions of Altair prior to 5: alt.Row('Row', header=alt.Header(title=None, labels=False))
Question: I am working with a Polars Series in Python and I need to obtain the number of unique values in the series. However, I want to exclude any null values from the result. For example, given the following Series: series = pl.Series([1, 2, None, 4, 2, 4, None])  The result should be 3 unique values (1, 2, and 4). What is the best way to achieve this in Polars?

Answer: Drop the nulls using .drop_nulls() and count the unique values using .n_unique(). Also, the correct way to represent null values in a call to pl.Series is using Python's None. pl.Null might have worked earlier but it doesn't in the latest polars. import polars as pl series = pl.Series([1, 2, None, 4, 2, 4, None]) print(series.drop_nulls().n_unique())  Output: 3
Question: I creat a signup form in django using django forms and when i run my code there is field i didnt expect Password-based authentication i did not use it and i have no idea what it is so anyone can tell me what it is and how i can remove it from user signup form?  form.py from django import forms from django.contrib.auth import get_user_model from django.contrib.auth.forms import UserCreationForm from django.contrib.auth.hashers import check_password class RegisterForm(UserCreationForm): """Form to Create new User""" def __init__(self, *args,hashed_code=None, **kwargs) -> None: super(RegisterForm,self).__init__(*args, **kwargs) self.hashed_code = hashed_code code = forms.CharField(max_length=4, required=True, label="code", help_text="Enter the four-digit code" ) def is_valid(self): """Return True if the form has no errors, or False otherwise.""" if not self.hashed_code: self.add_error("code","you have not any valid code get the code first") elif not check_password(self.data.get("code"),self.hashed_code) : self.add_error("code","code is invalid") return self.is_bound and not self.errors class Meta: model = get_user_model() fields = ["email", "password1", "password2","code"] 

Answer: That field comes from BaseUserCreationForm, the superclass of UserCreationForm. This is actually a regression in Django 5.1, and will be fixed when https://github.com/django/django/pull/18484 is released in Django 5.1.1. As a workaround, you should be able to delete the field from your subclass with class RegisterForm(UserCreationForm): usable_password = None # Workaround; see https://github.com/django/django/pull/18484
Question: I have two boolean arrays, first and second that should be mostly equal (up to a tolerance). I would like to compare them in a way that is forgiving if a few elements are different. Something like np.array_equal(first, second, equal_nan=True) is too strict because all values must be the same and np.allclose(first, second, atol=tolerance, equal_nan=True) is not suitable for comparing booleans. The following case should succeed: tolerance = 1e-5 seed = np.random.rand(100, 100, 100) first = seed > 0.5 second = (seed > 0.5) & (seed < 1. - 1e-6) # 99.9999% overlap in true elements  The following case should fail: first = seed > 0.5 second = (seed > 0.5) & (seed < 1. - 1e-4) # 99.99% overlap in true elements  The following case should also fail: first = seed > 0.5 second = first[::-1] # first.sum() == second.sum(), but they are not similar  How can I handle this case in an elegant manner?

Answer: Simon's answer was pretty close to what I needed. However, I preferred using a volume overlapping metric (eg dice and IoU) instead of normalizing by the size of the voxel. Dice and IoU range [0, 1], which is pretty convenient in this case, with 0 meaning no overlap and 1 meaning perfect overlap. Dice implementation: tolerance = 1e-5 dice = 2 * (first & second).sum() / (first.sum() + second.sum()) if 1 - dice > tolerance: raise  IoU implementation: iou = (first & second).sum() / (first | second).sum() if 1 - iou > tolerance: raise