eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to sort a matrix if I have [[5 7 1 2] [4 8 1 4] [4 0 1 9] [2 7 5 0]]  I want this result [[9 8 7 7] [5 5 4 4] [4 2 2 1] [1 1 0 0]]  this is my code, I'm using random numbers import numpy as np MatriX = np.random.randint(10, size=(4,4)) print(MatriX) for i in range(10): for j in range(10): a=np.sort(-MatriX,axis=1)*-1 b=np.sort(-a,axis=0)*-1 c=np.transpose(b) d=np.sort(-c,axis=1)*-1 e=np.sort(-d,axis=0)*-1 print(e)  but this code, don't work at all

Answer: It's as simple as this: import numpy as np data = np.random.randint(10, size=(4,4)) print(data) result = np.sort(data.flatten())[::-1].reshape(data.shape) print(result)  Output: [[4 7 7 6] [1 9 5 0] [5 8 1 2] [2 5 7 3]] [[9 8 7 7] [7 6 5 5] [5 4 3 2] [2 1 1 0]]  An explanation of the parts of np.sort(data.flatten())[::-1].reshape(data.shape):  data.flatten() flattens the array into a one-dimensional shape and returns the result np.sort() sorts the one-dimensional result and returns the sorted data [::-1] reverses the order of the sorted data .reshape(data.shape) applies the shape of the original data to that reversed sorted data  Note that something like this would be easier to read: result = data.flattened().sorted().reversed().reshaped(data.shape)  But numpy doesn't implement all the needed methods that allow for this more readable chaining and many methods make changes in-place. So you get the slightly harder to read solution above instead. (you could define a class that has these, but your code wouldn't make sense to others familiar with numpy, so it is probably best left the way it is) However, if your data is large, performing some operations in-place may make more sense. The solution given here is intended to be simple, you may want to look at some optimisation for very large data.
Question: I have been trying to implement in Python (with numpy and scipy) this variant of Matlab's corr function, but it seems I cannot solve it by myself. What I need is to implement the alternative Matlab corr implementation: [rho,pval] = corr(X,Y) I would appreciate any help on this!!! What I tried: I tried to modify the solutions posted here and in this other thread, without much success. For instance, I was able to hstack the two matrixes X and Y, and by keeping a part of the correlation matrix, I got the right result for the correlations. However, the same trick is NOT working for the p-values. Actually, both the solutions in the other threads gave me the (more or less) correct values for the correlation coefficients, which is great, but I cannot reproduce the behavior of the Matlab implementation for the p-values. Also, the solution here aims to reproduce corrcoef's behaviour, which, according to Matlab's documentation, converts the input matrices into column vectors before computing the correlations. On the other hand, I also tried to hstack the matrices in Matlab, and again got the same answer for the correlations, but the values are quite different for the p-values between what I get in Python and what I got in Matlab. This made me think that the problem perhaps is in the statistic being calculated. However, according to the Matlab documentation, it uses:  corr computes the p-values for Pearson's correlation using a Student's t distribution for a transformation of the correlation  and, from the documentation in SciPy I think they are using the same test, but I am not 100% sure, given that the bibligraphic reference there is for Student's paper, which is the one that Matlab's docs say it uses (Student's r), but as I said, I am NOT sure at all.

Answer: I think the easiest way is to do it in pandas, using scipy.stats pearsonr, which returns the pairwise rho and pval. I tested with some sample below, and I believe the results match the matlab results. import numpy as np from scipy.stats import pearsonr import pandas as pd X = np.array([ [0.5377, 0.3188, 3.5784, 0.7254], [1.8339, -1.3077, 2.7694, -0.0631], [-2.2588, -0.4336, -1.3499, 0.7147], [0.8622, 0.3426, 3.0349, -0.2050] ]) Y1 = np.array([ [-0.1241, 0.6715, 0.4889, 0.2939], [1.4897, -1.2075, 1.0347, -0.7873], [1.4090, 0.7172, 0.7269, 0.8884], [1.4172, 1.6302, -0.3034, -1.1471] ]) Y2 = Y1 Y2[:, 3] = Y2[:, 3] + X[:, 1] df1 = pd.DataFrame(X) df2 = pd.DataFrame(Y2) coeffmat = np.zeros((df1.shape[1], df2.shape[1])) pvalmat = np.zeros((df1.shape[1], df2.shape[1])) for i in range(df1.shape[1]): for j in range(df2.shape[1]): corrtest = pearsonr(df1[df1.columns[i]], df2[df2.columns[j]]) coeffmat[i,j] = corrtest[0] pvalmat[i,j] = corrtest[1] dfcoeff = pd.DataFrame(coeffmat, columns=df2.columns, index=df1.columns) print(dfcoeff) dfpvals = pd.DataFrame(pvalmat, columns=df2.columns, index=df1.columns) print(dfpvals)  I also tried playing around with pandas corrwith function (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corrwith.html#pandas.DataFrame.corrwith) but didn't spend enough time. I think there is a more elegant solution which uses that. Hope this helps
Question: I'm looking at how to use the pandas pct_change() function, but I need the values 'wrap around', so the last and first values create a percent change value in position 0 rather than NaN. For example: df = pd.DataFrame({'Month':[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], 'Value':[1, 0.9, 0.8, 0.75, 0.75, 0.8, 0.7, 0.65, 0.7, 0.8, 0.85, 0.9]}) Month Value 0 1 1.00 1 2 0.90 2 3 0.80 3 4 0.75 4 5 0.75 5 6 0.80 6 7 0.70 7 8 0.65 8 9 0.70 9 10 0.80 10 11 0.85 11 12 0.90  Using pct_change() + 1 gives: df['percent change'] = df['Value'].pct_change() + 1 Month Value percent change 0 1 1.00 NaN 1 2 0.90 0.900000 2 3 0.80 0.888889 3 4 0.75 0.937500 4 5 0.75 1.000000 5 6 0.80 1.066667 6 7 0.70 0.875000 7 8 0.65 0.928571 8 9 0.70 1.076923 9 10 0.80 1.142857 10 11 0.85 1.062500 11 12 0.90 1.058824  However I also need to know the % change between December (moth=12) and January (month=1), so the NaN should be 1.111111. I hope to eventually do this to several groups within a group by, so muddling about filling in the Nan with one value over the other, or manually calculating all the percentages seems a long winded way to do it. Is there a simpler way to achieve this?

Answer: Just use numpy.roll that is designed for this specific purpose: import numpy as np df['percent change'] = df['Value'].div(np.roll(df['Value'], 1))  Output:  Month Value percent change 0 1 1.00 1.111111 1 2 0.90 0.900000 2 3 0.80 0.888889 3 4 0.75 0.937500 4 5 0.75 1.000000 5 6 0.80 1.066667 6 7 0.70 0.875000 7 8 0.65 0.928571 8 9 0.70 1.076923 9 10 0.80 1.142857 10 11 0.85 1.062500 11 12 0.90 1.058824  If you need to perform this per group, combine it with groupby.transform: import numpy as np df['percent change'] = df['Value'].div(df.groupby('Group')['Value'].transform(lambda x: np.roll(x, 1)))  Output:  Month Value Group percent change 0 1 1.00 1 1.111111 1 2 0.90 1 0.900000 2 3 0.80 1 0.888889 3 4 0.75 1 0.937500 4 5 0.75 1 1.000000 5 6 0.80 1 1.066667 6 7 0.70 1 0.875000 7 8 0.65 1 0.928571 8 9 0.70 1 1.076923 9 10 0.80 1 1.142857 10 11 0.85 1 1.062500 11 12 0.90 1 1.058824 12 1 1.00 2 1.111111 13 2 0.90 2 0.900000 14 3 0.80 2 0.888889 15 4 0.75 2 0.937500 16 5 0.75 2 1.000000 17 6 0.80 2 1.066667 18 7 0.70 2 0.875000 19 8 0.65 2 0.928571 20 9 0.70 2 1.076923 21 10 0.80 2 1.142857 22 11 0.85 2 1.062500 23 12 0.90 2 1.058824
Question: With the code below: import abc class ABCParent(metaclass=abc.ABCMeta): def __init__(self, a: str, sibling_type: type[ABCParent]) -> None: self.a = a self._sibling_type = sibling_type def new_sibling(self, a: str) -> ABCParent: return self._sibling_type(a) class ChildA(ABCParent): def __init__(self, a: str) -> None: super().__init__(a, ChildB) class ChildB(ABCParent): def __init__(self, a: str) -> None: super().__init__(a, ChildA)  I get the following typing error, using mypy in strict mode. src/demo_problem.py: note: In member "new_sibling" of class "ABCParent": src/demo_problem.py:10:16:10:36: error: Missing positional argument "sibling_type" in call to "ABCParent" [call-arg] return self._sibling_type(a)  This is perfectly logical, since ABCParent requests the sibling_type argument. But in my case, the self._sibling_type will be one of the child classes, which override the constructor such that sibling_type is not needed. I was able to work around the problem by using a typing.Union of the child classes: import abc import typing TConcreteChild: typing.TypeAlias = typing.Union["ChildA", "ChildB"] class ABCParent(metaclass=abc.ABCMeta): def __init__(self, a: str, sibling_type: type[TConcreteChild]) -> None: ...  But I don't like this because:  It involves defining a type alias (TConcreteChild) which is otherwise useless The definition of this alias must be updated if more child classes are defined It involves defining types before definitions of the classes, so it requires forward references using strings (and therefore it can't use the new ChildA | ChildB syntax).  How can I do this more cleanly, while still having strict type checking?

Answer: So, I think one approach that could work depending on the specifics of your actual use-case is making ._sibling_type a generic callable and making the classes generic as well, parametrized with the same type variable. import abc import typing P = typing.TypeVar("P", bound="ABCParent") class ABCParent(typing.Generic[P], metaclass=abc.ABCMeta): def __init__(self, a: str, sibling_type: typing.Callable[[str], P]) -> None: self.a = a self._sibling_type = sibling_type def new_sibling(self, a: str) -> P: return self._sibling_type(a) class ChildA(ABCParent["ChildB"]): def __init__(self, a: str) -> None: super().__init__(a, ChildB) class ChildB(ABCParent["ChildA"]): def __init__(self, a: str) -> None: super().__init__(a, ChildA)  Types are callables. And in my experience with mypy, treating them like callables when you can is the easier approach. As an aside, you can still use the pipe operator syntax with string-based forward references, e.g.: def foo(x: "ChildA | ChildB") -> None: ...
Question: I have a data frame where the columns values are list and want to find the differences between two columns. data={'NAME':['JOHN','MARY','CHARLIE'], 'A':[[1,2,3],[2,3,4],[3,4,5]], 'B':[[2,3,4],[3,4,5],[4,5,6]]} df=pd.DataFrame(data)  Why doesn't it work? df = df.assign(X1 = lambda x: [y for y in x['A'] if y not in x['B']])  I get error :  TypeError: unhashable type: 'list'  I don't understand why?

Answer: So, this is where lambdas get interesting. These two lambdas will have the same result: df = df.assign(X1 = lambda x: [y for y in x['A']]) #unvectorized, x is the entire DataFrame df = df.assign(X1 = lambda x: x['A']) #vectorized, x is a single row  One (lengthy) way to do what you are asking is to iterate through each row, and then compare the nested lists: df = df.assign(X1 = lambda x: [[y for y in x['A'][i] if y not in x['B'][i]] for i in range(len(x['A']))])  which can be simplified to one of the following df = df.assign(X1 = [[y for y in r.A if y not in r.B] for i, r in df.iterrows()]) #similar structure to your initial solution df = df.assign(X2 = [list(set(r.A).difference(r.B)) for i, r in df.iterrows()]) #more efficient, especially for larger sets  Edit: Pulling in elements of @user19077881's answer, the row-wise operations do work if you use df.apply instead of df.assign df['X1'] = df.apply(lambda r: [y for y in r.A if y not in r.B], axis=1) df['X1'] = df.apply(lambda r: list(set(r.A).difference(r.B)), axis=1)
Question: I'm trying to terminate a Python program that uses many threads on its own. If I'm not mistaken, just sys.exit() works fine. However, to guard against my many mistakes, including losing references to threads, I tried the following: subprocess.Popen(['start', 'cmd.exe', '/c', f'timeout 5&taskkill /f /fi "PID eq {os.getppid()}"'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  I thought it was a problem with escaping the quotes, so I tried several things but failed. I gave up and did the following and it worked perfectly. with open('exit_self.bat', 'w') as file: file.write(f'timeout 5&taskkill /f /fi "PID eq {os.getppid()}"&del exit_self.bat') subprocess.Popen(['start', 'cmd.exe', '/c', 'exit_self.bat'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  How can I do it without temp files? What did I miss? For reference, I used /k instead of /c option of cmd.exe to leave the window and check the error message in the window, it is as follows: Waiting for 0 seconds, press a key to continue ... ERROR: Invalid argument/option - 'eq'. Type "TASKKILL /?" for usage.  I'm not sure if it will help, but I added echo to see the syntax of the command being executed: subprocess.Popen(['start', 'cmd.exe', '/k', 'echo', f'timeout 5&taskkill /f /fi "PID eq {os.getppid()}"'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  The result is: "timeout 5&taskkill /f /fi \"PID eq 3988\"" 

Answer: From subprocess.Popen:  args should be a sequence of program arguments or else a single string or path-like object.  Run notepad.exe and be sure to not type in its window so "Untitled - Notepad" is the title. I'm using WINDOWTITLE eq Untitled* so I don't have to look up the PID. Use a single string for the command. This will open a window, countdown, and kill Notepad. Note that the entire /c parameter must be quoted so it is passed to the second command window. import subprocess import time p = subprocess.Popen('start cmd.exe /c "timeout 5&taskkill /f /fi "WINDOWTITLE eq Untitled*""', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) for i in range(3): print(str(i) * 10) time.sleep(1)  Output in original command window showing parallel work. Another cmd window opens with the countdown. Notepad was killed 2 seconds after 2222222222 printed due to the 5 second total timeout before taskkill: 0000000000 1111111111 2222222222
Question: Say for example I have data like this: import polars as pl df = pl.DataFrame( { "subject": ["subject1", "subject2"], "emails": [ ["samATxyz.com", "janeATxyz.com", "jimATcustomer.org"], ["samATxyz.com", "zaneATxyz.com", "basATcustomer.org", "jimATcustomer.org"], ], } ) df  shape: (2, 2) ┌──────────┬─────────────────────────────────────────────────────────────────────────────┐ │ subject ┆ emails │ │ --- ┆ --- │ │ str ┆ list[str] │ ╞══════════╪═════════════════════════════════════════════════════════════════════════════╡ │ subject1 ┆ ["samATxyz.com", "janeATxyz.com", "jimATcustomer.org"] │ │ subject2 ┆ ["samATxyz.com", "zaneATxyz.com", "basATcustomer.org", "jimATcustomer.org"] │ └──────────┴─────────────────────────────────────────────────────────────────────────────┘  I want to filter the data so that the emails column only contain emails that end in "ATxyz.com". shape: (2, 2) ┌──────────┬───────────────────────────────────┐ │ subject ┆ emails │ │ --- ┆ --- │ │ str ┆ list[str] │ ╞══════════╪═══════════════════════════════════╡ │ subject1 ┆ ["samATxyz.com", "janeATxyz.com"] │ │ subject2 ┆ ["samATxyz.com", "zaneATxyz.com"] │ └──────────┴───────────────────────────────────┘  How can I do this using polars? I had a few ideas, but I cannot figure out the right syntax, or it seems more complex/verbose than I would expect:  Maybe I could somehow filter the data using .list.eval(pl.element() ..., but I cannot figure out how to filter items in the list with this syntax. I could reshape the data using .explode, but this seems verbose and more complex than needed.  This is as close as I have got import polars as pl df = pl.DataFrame( { "subject": ["subject1", "subject2"], "emails": [ ["samATxyz.com", "janeATxyz.com", "jimATcustomer.org"], ["samATxyz.com", "zaneATxyz.com", "basATcustomer.org", "jimATcustomer.org"], ], } ) df.with_columns( pl.col("emails").list.eval(pl.element().str.contains("ATxyz")), )  shape: (2, 2) ┌──────────┬────────────────────────────┐ │ subject ┆ emails │ │ --- ┆ --- │ │ str ┆ list[bool] │ ╞══════════╪════════════════════════════╡ │ subject1 ┆ [true, true, false] │ │ subject2 ┆ [true, true, false, false] │ └──────────┴────────────────────────────┘ 

Answer: You were on the right track with pl.Expr.list.eval. It can be combined with pl.Expr.filter to achieve the desired result as follows. df.with_columns( pl.col("emails").list.eval( pl.element().filter(pl.element().str.ends_with("ATxyz.com")) ) )  shape: (2, 2) ┌──────────┬───────────────────────────────────┐ │ subject ┆ emails │ │ --- ┆ --- │ │ str ┆ list[str] │ ╞══════════╪═══════════════════════════════════╡ │ subject1 ┆ ["samATxyz.com", "janeATxyz.com"] │ │ subject2 ┆ ["samATxyz.com", "zaneATxyz.com"] │ └──────────┴───────────────────────────────────┘
Question: The line demo_code = print("foo bar") # some comments and this line length exceed 79 that i config.  is being formatted by ruff format to demo_code = print( "foo bar" ) # some comments and this line length exceed 79 that i config.  as it exceeds the value defined in line-length. I want to make ruff count only command characters and ignore the comments.

Answer: The best (unique?) solution to ignore lengthy code line errors (E501) with ruff is using codetags in you code: # pyproject.toml [tool.ruff.lint.pycodestyle] ignore-overlong-task-comments = true [tool.ruff.lint] task-tags = ["HACK"]  demo_code = print("foo bar") # HACK: That is a so savvy solution. Now, I can write very lengthy inline comments without any problem :)
Question: Is there an idiomatic way to get specific rows from a LazyFrame? There's two methods I could figure out. Not sure which is better, or if there's some different method I should use. import polars as pl df = pl.DataFrame({"x": ["a", "b", "c", "d"]}).lazy() rows = [1, 3] # method 1 ( df.with_row_index("row_number") .filter(pl.col("row_number").is_in(rows)) .drop("row_number") .collect() ) # method 2 df.select(pl.all().gather(rows)).collect() 

Answer: pl.Expr.gather is the idiomatic way to take values by index. df.select(pl.all().gather(rows)).collect()   For completeness, method 1 can be refined by using an expression for the index. This way no temporary column is created and dropped again. # method 1.1 df.filter(pl.int_range(pl.len()).is_in(rows)).collect()
Question: I have two chains, one that generates a document and one that creates a short document resume. I want to chain them, using the output from the first on inside the other one. But I want to get both outputs in the result. Before LCEL, I could do it using LLMChain's output_key parameter. With LCEL, there seems to be a RunnablePassthrough class, but I don't seem to get how to use it to aggregate the output. Code example: generate_document_chain = generate_document_prompt | llm | StrOutputParser() resume_document_chain = resume_document_prompt | llm | StrOutputParser() aggregated_chain = generate_document_chain | resume_document_chain content = aggregated_chain.invoke({"topic": topic}) 

Answer: Perhaps the following is what you want. It feeds the output of the first chain into second chain as input. from langchain_core.runnables import RunnablePassthrough aggregated_chain = generate_document_chain | { "first_chain_output": RunnablePassthrough(), "second_chain_output": resume_document_chain } content = aggregated_chain.invoke({"topic": topic})  Then the output will be a dictionary with keys: "first_chain_output" and "second_chain_output". You can also use RunnablePassthrough.assign. Unlike the case above, the key of generate_document_chain output should match the input variable name of the second chain. Below, the input variable of the second chain is assumed to be "input" (btw, the input variable of the first chain is "topic"). aggregated_chain = ( {"input": generate_document_chain} | RunnablePassthrough.assign(second_chain_output=resume_document_chain) )  The output of this chain will be a dict with keys: "input" and "second_chain_output".
Question: An existing client is already sending data in a structure like… { "hive_metadata": {"name": "hive name"}, "bees": [{"name": "bee 1", "name": "bee 2", ...}] }  For models like: class Hive(models.Model): name = models.CharField(max_length=32, help_text="name") class Bee(models.Model): name = models.CharField(max_length=32, help_text="name") hive = models.ForeignKey( Hive, help_text="The Hive associated with this Bee", on_delete=models.CASCADE )  The code that makes this possible manually iterates over the incoming data. I would like to rewrite it using a django rest framework serializer; however, the fact that hive_metadata is nested itself has stumped me so far. If I write class BeesSerializer(ModelSerializer): class Meta: model = models.Bee fields = ("name",) class PopulatedHiveSerializer(ModelSerializer): bees = BeesSerializer(many=True, source="bee_set") class Meta: model = models.Hive fields = ("name","bees",)  would produce { "name": "hive name", "bees": [{"name": "bee 1", "name": "bee 2", ...}] }  readily enough. I had hoped I could solve it with a reference to a sub-serializer, something like class HiveMetaDataSerializer(ModelSerializer): class Meta: model = models.Hive fields = ("name",) class PopulatedHiveSerializer(ModelSerializer): bees = BeesSerializer(many=True, source="bee_set") hive_metadata = HiveMetaDataSerializer(source=???) class Meta: model = models.Hive fields = ("hive_metadata","bees",)  but I can't seem to figure out what to put in the "source" so that the same object is passed through the outer serializer into the inner. So, is there a way to do this using a django rest framework serializer?

Answer: You can use a string literal with an asterisk ('*'), as specified in the documentation [drf-doc]:  The value source='*' has a special meaning, and is used to indicate that the entire object should be passed through to the field. This can be useful for creating nested representations, or for fields which require access to the complete object in order to determine the output representation.  so we can use: class PopulatedHiveSerializer(ModelSerializer): bees = BeesSerializer(many=True, source='bee_set') hive_metadata = HiveMetaDataSerializer(source='*') class Meta: model = models.Hive fields = ( 'hive_metadata', 'bees', )