eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: If I run this code through mypy: if __name__ == '__main__': list1: list[str] = [] list2: list[str | int] = [] list2 = list1  It gives me following error: error: Incompatible types in assignment (expression has type "List[str]", variable has type "List[Union[str, int]]")  Why? Isn't a list[str] a subset of list[str | int]? If yes, then why can't I assign list1 which doesn't have wider range of possible types to list2?

Answer: Generic types can be classified as covariant, contravariant, or invariant. When u is a subtype of v, a generic type g is  covariant when g[u] is a subtype of g[v], contravariant when g[v] is a subtype of g[u], or invariant when neither g[u] nor g[v] is a subtype of the other.  list is invariant because lists are mutable, in which case it is not safe to substitute one concrete type of list for another. Immutable sequences are covariant: Sequence[str] is, indeed, a subtype of Sequence[str | int], so you could do the following: from collections.abc import Sequence if __name__ == '__main__': list1: Sequence[str] = [] list2: Sequence[str | int] = [] list2 = list1  The only thing you can do with a Sequence, regardless of whether it is a list, tuple, etc, is read from it, and anything reading from list2 will know it can't assume that any given element is a str or an int (though it will be one or the other, rather than, say, a float), so it's safe to provide a sequence of either type.
Question: I have the below dataframe     salesman north_access south_access east_access west_access     A 1 0 0 1   B 0 1 1 1   C 1 0 1 1     I want to convert the above into the below format     salesman direction access     A north 1   A south 0   A east 0   A west 1   B north 0   B south 1   B east 1   B west 1     I tried exploring the split and transpose function but didnt get the expected results. Can someone please help with the code to make the above changes in python, thanks in advance.

Answer: Another solution (using pd.wide_to_long): df.columns = [f'access_{c.split("_")[0]}' if "_access" in c else c for c in df.columns] x = pd.wide_to_long( df, stubnames="access", suffix=r".*", i=["salesman"], j="direction", sep="_" ).reset_index() print(x)  Prints:  salesman direction access 0 A north 1 1 B north 0 2 C north 1 3 A south 0 4 B south 1 5 C south 0 6 A east 0 7 B east 1 8 C east 1 9 A west 1 10 B west 1 11 C west 1
Question: I am trying to implement my custom optimizer in PyTorch or TensorFlow 2.12.0. With help of ChatGPT I always get code that have errors, what's more I can't find any useful examples. I would like to implement custom optimizer as: d1 contains sign of current derivatives d2 contains sign of previous derivatives step_size is 1.0 step_size is divided by 2.0 if sign of d1 != d2 In PyTorch I know that code has to look something like this: import torch.optim as optim class MyOpt(optim.Optimizer): def __init__(self, params, lr=1.0): defaults = dict(lr=lr, d1=None, d2=None) super(MyOpt, slef).__init__(params, defaults) def step(self): ???  Can anyone help me to code it ?

Answer: I did it. class MyOptimizer(optim.Optimizer): def __init__(self, params, lr=1.0): defaults = dict(lr=lr) super(MyOptimizer, self).__init__(params, defaults) self.lr = {} self.d2 = {} for group in self.param_groups: for param in group['params']: self.lr[param] = torch.ones_like(param.data) * lr self.d2[param] = torch.ones_like(param.data) def step(self, closure=None): loss = None if closure is not None: loss = closure() for group in self.param_groups: for param in group['params']: if param.grad is None: continue d1 = torch.sign(param.grad.data) t = torch.where(torch.sign(self.d2[param]) == d1, 1.0, 2.0) lr = self.lr[param] lr = lr / t self.lr[param] = lr param.data -= d1 * lr self.d2[param] = d1 return loss
Question: I need to parse a Pubchem database to search for certain clues on the pages of compounds (Toxicity codes, to be exact, they look like 'H300'), and then add their CIDs to the correspondent lists The Database is here https://ftp.ncbi.nih.gov/pubchem/Compound/CURRENT-Full/XML/ But the xml.gz files there are so big that they can't be unpacked on my computer So maybe there is a way to read this files directly on the server of a PubChem

Answer: One way I would approach this is to use curl and gunzip and maybe grep: Example: curl -ks https://ftp.ncbi.nih.gov/pubchem/Compound/CURRENT-Full/XML/Compound_000000001_000500000.xml.gz -o - | gunzip | grep someString  This will stream down the file, and in realtime decompress it, which will allow you in realtime to grep for what you need
Question: If you consider inner_multiply as an initializer of multiply, shouldn't you make them loosely coupled and DI the initializer (or any other way) especially if you require multiple initializers? Or am I misunderstanding the basic concept of curried function in FP? def inner_multiply(x): def multiply(y): return x * y return multiply def curried_multiply(x): return inner_multiply(x) multiply_by_3 = curried_multiply(3) result = multiply_by_3(5) print(result) # Output: 15 (3 * 5) 

Answer: You can define an entirely generic curry function like this: def curry(f): return lambda x: lambda y: f(x, y)  Assume, in order to reproduce the example in the OP, that you also define a multiply function like this: def multiply(x, y): return x * y  You can now partially apply multiply using curry: >>> multiply_by_3 = curry(multiply)(3) >>> multiply_by_3(5) 15  This example immediately uses multiply_by_3, but the benefit is that you don't have to do that. Rather, you can partially apply the function in one place, pass that partially applied function around, and call it in an entirely different part of your code base.
Question: A class Message extends Enum to add some logic. The two important parameters are verbose level and message string, with other optional messages (*args). Another class MessageError is a special form of the Message class in which verbose level is always zero, everything else is the same. The following code screams TypeError:  TypeError: Enum.new() takes 2 positional arguments but 3 were given  from enum import Enum class Message(Enum): verbose: int messages: list[str] def __new__(cls, verbose: int, message: str, *args): self = object.__new__(cls) self._value_ = message return self def __init__(self, verbose: int, message: str, *args): self.verbose = verbose self.messages = [self.value] + list(args) class MessageError(Message): def __new__(cls, message: str): return super().__new__(cls, 0, message) # <- problem is here! def __init__(self, message: str): return super().__init__(0, message) class Info(Message): I_001 = 2, 'This is an info' class Error(MessageError): E_001 = 'This is an error'  I was expecting that super().__new__(cls, 0, message) would call __new__ from the Message class, but it seems that is not the case. What am I doing wrong here?

Answer: Enums are unusual (aka weird) in several regards, with creation being the biggest area. During enum class creation, after the members themselves have been created but before the class is returned, any existing __new__ is renamed to __new_member__1 and the __new__ from Enum itself is inserted into the class -- this is so calls like Info('This is an error') will return the existing member (or raise), and not create a new member. Your MessageError.__new__ should look like:  def __new__(cls, message: str): return super().__new_member__(cls, 0, message)   1 In 3.11+ it's also saved as _new_member_ to follow the pattern of sunder names for enum-specific methods and attributes.  Disclosure: I am the author of the Python stdlib Enum, the enum34 backport, and the Advanced Enumeration (aenum) library.
Question: I build an Flask API endpoint which when called will execute an action to process some data (the data are not send but come from the database). The result of the method(action) called is to update some fields in some database objects based on calculations performed on the objects themselves. Initially from what I found most people use POST requests, but post request is mainly designed for sending data in order to create a resource and in these cases no such thing takes place. I looked at all the possible HTTP request methods and no one fits the bill of just asynchronously executing some code in the server without sending and returning any data. Am I missing something, should I simply use POST or is there a better alternative?

Answer: PATCH seems to be the correct method for your use case since you're updating some fields of a resource, although POST should also be fine. You can use 204 status code (which means api was successfully executed but there's no content to send) if you're updating the fields before api return. Refer this: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/204 But if you're updating the fields async after api return, you should send 202 status code (which means request was accepted but not yet processed): https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/202
Question: I have a DF as below. I have added a new column which has Total of all the rows and a new row which has total of all columns: A B C D Total ------------------- 1 2 3 4 10 5 6 7 8 26 6 8 10 12 36  Now I need to add one more row for which the first element will be NaN and rest will be a column subtracted from the previous column in the Total row. A B C D Total 1 2 3 4 10 5 6 7 8 26 6 8 10 12 36 NaN 2 2 2 24 <--- new row  Thanks

Answer: This would be one of the rare use cases for df.append, but in lieu of that you can extract the last row with iloc[-1] and diff the individual values, then combine this back with the original. Option 1 One method of doing this concatenation would be using pd.concat df2 = pd.concat([df, df.iloc[-1].diff().to_frame().T]) print (df2) A B C D Total 0 1.0 2.0 3.0 4.0 10.0 1 5.0 6.0 7.0 8.0 26.0 2 6.0 8.0 10.0 12.0 36.0 2 NaN 2.0 2.0 2.0 24.0  Where, df.iloc[-1].diff().to_frame().T # dataframe with 1 row A B C D Total 2 NaN 2.0 2.0 2.0 24.0   Option 2 An alternative using inplace assignment with loc: df.loc[len(df.index)] = df.iloc[-1].diff() print (df) A B C D Total 0 1.0 2.0 3.0 4.0 10.0 1 5.0 6.0 7.0 8.0 26.0 2 6.0 8.0 10.0 12.0 36.0 3 NaN 2.0 2.0 2.0 24.0  Where, df.iloc[-1].diff() # series A NaN B 2.0 C 2.0 D 2.0 Total 24.0 Name: 2, dtype: float64   Option 3 Here's an option that has a bit of fun with dictionaries and pd.DataFrame: pd.DataFrame([*df.to_dict('records'), df.iloc[-1].diff().to_dict()]) A B C D Total 0 1.00 2.00 3.00 4.00 10.00 1 5.00 6.00 7.00 8.00 26.00 2 6.00 8.00 10.00 12.00 36.00 3 NaN 2.00 2.00 2.00 24.00   Option 4 [deprecated] On older versions (pandas <= 1.4) I would have recommended using append like this: df2 = df.append(df.iloc[-1].diff(), ignore_index=True)
Question: I have a DataFrame like this: import pandas as pd import numpy as np df = pd.DataFrame({ "realization_id": np.repeat([0, 1], 6), "sample_size": np.tile([0, 1, 2], 4), "num_obs": np.tile(np.repeat([25, 100], 3), 2), "accuracy": [0.8, 0.7, 0.8, 0.6, 0.7, 0.5, 0.6, 0.7, 0.8, 0.7, 0.9, 0.7], "prob": [0.94, 0.96, 0.95, 0.98, 0.93, 0.92, 0.90, 0.92, 0.95, 0.9, 0.91, 0.92] }) df["accum_max_prob"] = df.groupby(["realization_id", "num_obs"])["prob"].cummax()  And I want to know how to create a column with this output: df["desired_accuracy"] = [0.8, 0.7, 0.7, 0.6, 0.6, 0.6, 0.6, 0.7, 0.8, 0.7, 0.9, 0.7]  Each entry of desired_accuracy equals the accuracy value that corresponds to the row where the highest prob has been achieved so far by group (that is why I create accum_max_prob). So, for example: the first value is 0.8 because there is no data prior to that, but then the next one is 0.7 because the prob of the second row is greater than the first. The third value stays the same, because the third prob is lower than the second one, so it does not update desired_accuracy. For each pair (realization_id, num_obs) the criteria resets. How can I do it in a vectorized fashion using Pandas?

Answer: It looks like: df['desired_accuracy'] = df['accuracy'].mask(df['prob'].lt(df['accum_max_prob'])).ffill()  Output:  realization_id sample_size num_obs accuracy prob accum_max_prob desired_accuracy 0 0 0 25 0.8 0.94 0.94 0.8 1 0 1 25 0.7 0.96 0.96 0.7 2 0 2 25 0.8 0.95 0.96 0.7 3 0 0 100 0.6 0.98 0.98 0.6 4 0 1 100 0.7 0.93 0.98 0.6 5 0 2 100 0.5 0.92 0.98 0.6 6 1 0 25 0.6 0.90 0.90 0.6 7 1 1 25 0.7 0.92 0.92 0.7 8 1 2 25 0.8 0.95 0.95 0.8 9 1 0 100 0.7 0.90 0.90 0.7 10 1 1 100 0.9 0.91 0.91 0.9 11 1 2 100 0.7 0.92 0.92 0.7
Question: So, I've done a lot of research on this and saw various other links and conferred with the Python documentation, however, I'm still a bit unclear on this. Maybe the way I see classes in Python is a bit wrong. As I understand, keys in a dictionary must be immutable. However, classes can be keys in a dictionary because of their default hash implementation (I think?). Why is this the case since classes are mutable? For example, class C: def __init__(self): self.val = 15 self.array = [] c = C() D = {c: 15} c.val = 14 c.array.append(15) print(D[c])  Why is this okay?

Answer: Instances of your C class are actually hashable, it comes with a default implementation of __hash__ which pertains to the identity of that object: >>> hash(c) # calls c.__hash__() 306169194  This __hash__ implementation allows your instance to be used as a key in a dictionary. This explains "Why doesn't changing things in your class change the hash?" — because the identity/reference of the instance doesn't change even if its contents do. On older versions of python, this used to be exactly equal to the object's id, but from python 3 onwards, it seems to be some derivative of it. This post goes into the gory details on the hash implementation.  Now let's say you wanted to prevent instances of your class from being used as a key... you could then do this (from the documentation):  If a class that does not override __eq__() wishes to suppress hash support, it should include __hash__ = None in the class definition.  class C: def __init__(self): self.val = 15 self.array = [] __hash__ = None # suppressing hash support c = C()  And now you get a familiar sounding TypeError if you attempt to retrieve the hash of c: >>> hash(c) # TypeError: unhashable type: 'C'  Naturally, this also implies you cannot use c as a dictionary key anymore (IOW trying to initialize {c: 15} would also throw the same TypeError since c is not hashable).
Question: (Extracted from another question.) Removing this set's 200,000 elements one by one like this takes 30 seconds (Attempt This Online!): s = set(range(200000)) while s: for x in s: s.remove(x) break  Why is that so slow? Removing a set element is supposed to be fast.

Answer: I think this is happening because you are removing the first element in the set every time. This creates a set which is increasingly empty one each iteration, so each time you create a new iterator and call __next__, it has to search further and further away. So, here is the source code for the iterator __next__ It has to find the next entry like this: while (i <= mask && (entry[i].key == NULL || entry[i].key == dummy)) i++;  The the iterator __next__ works by finding the first non-empty, non-dummy value: So, say we have something like: entries = [null, 1, null, 2, null, 3, null, 4, null, 5]  Then on each iteration of the while loop, you get: entries = [null, 1, null, 2, null, 3, null, 4, null, 5] entries = [null, DUMMY, null, 2, null, 3, null, 4, null, 5] entries = [null, DUMMY, null, DUMMY, null, 3, null, 4, null, 5] entries = [null, DUMMY, null, DUMMY, null, DUMMY, null, 4, null, 5]  So each time, the iterator has to search further and further away from the beginning of the entires, since each iteration of the while loop removes the first one. Hence, the quadratic time behavior.
Question: df = pd.DataFrame({ 'caseid': [1, 1, 1, 2, 2, 2, 3, 3, 3], 'timestamp': [10, 20, 30, 10, 20, 30, 10, 20, 30] 'var1': [np.nan, np.nan, np.nan, 10, np.nan, 11, 12, 13, 14], 'var2': [2., 3., 4., np.nan, 5., 6., np.nan, np.nan, np.nan] })  I need to find the first (and last) valid timestamp for each variable per caseid. I.e. For var1, caseid 1 it would be None, for caseid 2 it would be 10 (last 30). And the same for each additional var column. Is there handle groups of ids without looping over caseid and doing a first_valid_index() on each column, since loops not most efficient when using pandas?

Answer: You can select the desired columns with filter (or manually), then replace the non-NA values with the timestamp (using mul and where), finally use a groupby.agg with first/last: m = df.filter(like='var').notna() out = (m.mul(df['timestamp'], axis=0).where(m) .groupby(df['caseid']).agg(['first', 'last']) )  Output:  var1 var2 first last first last caseid 1 NaN NaN 10.0 30.0 2 10.0 30.0 20.0 30.0 3 10.0 30.0 NaN NaN  Intermediate: m.mul(df['timestamp'], axis=0).where(m) var1 var2 0 NaN 10.0 1 NaN 20.0 2 NaN 30.0 3 10.0 NaN 4 NaN 20.0 5 30.0 30.0 6 10.0 NaN 7 20.0 NaN 8 30.0 NaN
Question: I am subclassing a dict and would love some help understanding the behavior below (please) [Python version: 3.11.3]: class Xdict(dict): def __init__(self, d): super().__init__(d) self._x = {k: f"x{v}" for k, v in d.items()} def __getitem__(self, key): print("in __getitem__") return self._x[key] def __str__(self): return str(self._x) def __iter__(self): print("in __iter__") d = Xdict({"a": 1, "b": 2}) print(d) print(dict(d))  Produces this output: {'a': 'x1', 'b': 'x2'} in __getitem__ in __getitem__ {'a': 'x1', 'b': 'x2'}  If I comment out the __iter__ method the output changes like so: {'a': 'x1', 'b': 'x2'} {'a': 1, 'b': 2}  Obviously the __iter__ method is not getting called, however its presence is affecting the behaviour. I am just interested in why this happens. I am not looking for alternative solutions to prevent it. Thanks, Paul.

Answer: Python's internals often directly invoke the C-level implementations of built-in class functionality, even in cases where a subclass may have overridden that functionality, leading to a lot of weird bugs where method overrides aren't invoked where you'd expect them to be. This is the case for a lot of the dict implementation, but when dicts became order-preserving in Python 3.6, one of those bugs hit the standard library: when x is an OrderedDict, dict(x) would copy the underlying dict implementation's order, instead of the OrderedDict order (which is tracked separately). To fix this bug, they added a check to dict_merge, in the code where it decides whether to use the fast path: if (PyDict_Check(b) && (Py_TYPE(b)->tp_iter == (getiterfunc)dict_iter)) {  dict_merge is the underlying routine responsible for copying the contents of another mapping into a dict. Previously, this line just said if (PyDict_Check(b)) {, which would use the fast path if the other mapping was any dict instance. Now, it also checks that the instance doesn't have an overridden __iter__. If the instance has an overridden __iter__, dict_merge will use the slow path, hence the difference you saw. However, the slow path doesn't actually use __iter__. It uses keys, which is why your code worked even though your __iter__ doesn't return an iterator.
Question: I have a large Pandas DataFrame (~800M rows), which I have indexed on a MultiIndex with two indices, an int and a date. I want to retrieve a subset of the DataFrame's rows based on a list of ints (about 10k) that I have. The ints match the first index of the multi-index. The multi-index is unique. The first thing I tried is to sort the index and then query it using loc: df = get_my_df() # 800M rows ids = [...] # 10k ints, sorted list df.set_index(["int_idx", "date_idx"], inplace=True, drop=False) df.sort_index(inplace=True) idx = pd.IndexSlice res = df.loc[idx[ids, :]]  However this is painfully slow, and I stopped running the code after about an hour. Next thing I tried was to set only the first one as index. This is suboptimal for me because the index is not unique, and also later I'll need to to further filter by date: df.set_index("int_idx", inplace=True, drop=False) df.sort_index(inplace=True) idx = pd.IndexSlice res = df.loc[idx[ids, :]]  To my surprise this was an improvement, but still very slow. I have two questions:  How can I make my query faster? (Either using single index or multi-index) Why is a sorted multi-index still so slow? 

Answer: It can be difficult to retrieve a subset of a DataFrame containing 800M rows. Here are some ideas to help your search go more quickly:  Use .loc() with boolean indexing instead of pd.IndexSlice:  Use boolean indexing with.loc() instead of pd.IndexSlice to slice your multi-index instead. This can assist Pandas in avoiding the costly practise of establishing a new index object for each slice when working with huge DataFrames. For example: res = df.loc[df.index.get_level_values('int_idx').isin(ids)]   Avoid setting the index multiple times:  It can be costly to set the index and sort the data numerous times. Try to just set the index once if you can, but try to avoid sorting it. For example: df.set_index(["int_idx", "date_idx"], inplace=True, drop=False) res = df[df.index.get_level_values('int_idx').isin(ids)]   Use chunking or parallel processing:  You might want to think about dividing your DataFrame into smaller parts, processing each one separately, and then concatenating the results if it is too big to store in memory. To speed up the query, you might also use parallel processing. Both of these tactics work well with the Dask library. In response to your second query, a sorted multi-index ought should be quicker than an unsorted one because it enables Pandas to utilise the quick search methods built into NumPy. However, if a huge DataFrame has numerous columns or the sorting order is complicated, sorting the data can be expensive. Generally speaking, sorting a DataFrame is an expensive process that should be avoided wherever possible.
Question: Using reserved keywords or built-in functions as variable/attribute names is commonly seen as bad practice. However, the SQLALchemy tutorial is full of exampled with attributes named id. Straight from the tutorial >>> class User(Base): ... __tablename__ = "user_account" ... ... id: Mapped[int] = mapped_column(primary_key=True) ... name: Mapped[str] = mapped_column(String(30)) ... fullname: Mapped[Optional[str]] ... ... addresses: Mapped[List["Address"]] = relationship( ... back_populates="user", cascade="all, delete-orphan" ... ) ... ... def __repr__(self) -> str: ... return f"User(id={self.id!r}, name={self.name!r}, fullname={self.fullname!r})"  Why is it not recommended to use id_ instead, as recommended at least for keywords in PEP 8?

Answer: Firstly, id is not a keyword; if it was one, you would get a syntax error (try replacing id there with e.g. pass). It is a global identifier for a built-in function. And while it is a bad practice to clobber over the default identifiers defined by Python, note that id: Mapped[int] = mapped_column(primary_key=True) inside a class definition defines a static attribute of that class, not a global variable. The global function id is not the same as the static attribute User.id (or self.id, as it is used later) — there is no conflict there. class Foo: id = 42 print(Foo.id) # => 42 print(id) # => <built-in function id>  However, as you correctly note: id = "YOU SHOULDN'T DO THIS" print(id) # => YOU SHOULDN'T DO THIS # (and not <built-in function id>)  Secondly, as to why id is used: SQLAlchemy maps classes to SQL tables with the snakecased class name, and attributes to SQL columns with the as-is attribute name (unless you specifically say id_: mapped_column('id', primary_key=True)), so id and id_ would be different columns in SQL. id (as opposed to id_) is the conventional name of the primary key column in SQL (if there is a single-column primary key). Some people would name the column user_id, but many dislike having user_id in table called users, since it should be obvious what users.id, and so users.user_id is redundant; the form user_id is generally reserved for foreign keys. The SQLAlchemy examples merely follow this SQL convention.