eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to be able to do something like: foo = Counter(['bar', 'shoo', 'bar']) tmp = {} tmp[foo] = 5  In other words, is there a hashable equivalent for Counter? Note that I can't use frozenset since I have repeated elements that I want to keep in the key. Edit: In my actual application, the objects in foo may not be comparable with each other so the list cannot be sorted.

Answer: What you seem to require is a way to use unordered pairs of key-amount as keys. A frozenset is probably the way to go, although you will have to create it out of the items of the Counter and not its keys. foo = Counter(['bar', 'shoo', 'bar']) tmp = {} tmp[frozenset(foo.items())] = 5 # tmp: {frozenset({('bar', 2), ('shoo', 1)}): 5}  If this is satisfying, you could implement this transformation by defining you own mapping type like so: from collections import Counter class CounterDict: def __init__(self): self.data = {} def __setitem__(self, key, value): if isinstance(key, Counter): self.data[frozenset(key.items())] = value else: raise TypeError def __getitem__(self, key): if isinstance(key, Counter): return self.data[frozenset(key.items())] else: raise TypeError foo = Counter(['bar', 'shoo', 'bar']) tmp = CounterDict() tmp[foo] = 42 tmp[foo] # 42  You could make this implementation richer by making CounterDict a subclass of collections.UserDict.
Question: I have a token that includes the following header eyJraWQiOiI4NkQ4OEtmIiwiYWxnIjoiUlMyNTYifQ. How can I obtain the following JSON decoding of it as jwt.io provides? { "kid": "86D88Kf", "alg": "RS256" }  jwt.decode() doesn't give this header. Thanks!

Answer: This is an unencrpyted header. Its a URL-safe base64 encoding of a JSON encoding of the data you want. You need to add padding characters to the end of the encoded string to make sure its on a 4 character boundary, then decode. >>> import json >>> import base64 >>> token = "eyJraWQiOiI4NkQ4OEtmIiwiYWxnIjoiUlMyNTYifQ" >>> padded = token + "="*divmod(len(token),4)[1] >>> padded 'eyJraWQiOiI4NkQ4OEtmIiwiYWxnIjoiUlMyNTYifQ==' >>> jsondata = base64.urlsafe_b64decode(padded) >>> jsondata b'{"kid":"86D88Kf","alg":"RS256"}' >>> data = json.loads(jsondata) >>> data {'kid': '86D88Kf', 'alg': 'RS256'}
Question: My data file looks like this: data.txt user,activity,timestamp,x-axis,y-axis,z-axis 0,33,Jogging,49105962326000,-0.6946376999999999,12.680544,0.50395286; 1,33,Jogging,49106062271000,5.012288,11.264028,0.95342433; 2,33,Jogging,49106112167000,4.903325,10.882658000000001,-0.08172209; 3,33,Jogging,49106222305000,-0.61291564,18.496431,3.0237172;  As can be seen, the last column ends with a semicolon, so when I read into pandas, the column is inferred as type object (ending with the semicolon. df = pd.read_csv('data.txt') df user activity timestamp x-axis y-axis z-axis 0 33 Jogging 49105962326000 -0.694638 12.680544 0.50395286; 1 33 Jogging 49106062271000 5.012288 11.264028 0.95342433; 2 33 Jogging 49106112167000 4.903325 10.882658 -0.08172209; 3 33 Jogging 49106222305000 -0.612916 18.496431 3.0237172;  How do I make pandas ignore that semicolon?

Answer: The problem with your txt is that it has mixed content. As I can see the header doesn't have the semicolon as termination character If you change the first line adding the semicolon it's quite simple pd.read_csv("data.txt", lineterminator=";")
Question: I came across function with signature like this: def get_quantile(numbers: List[float], q: float | int ) -> float | int | None :  What does it mean? It's a syntax error on my python 3.8. Do I need to import something from future to make it work?

Answer: According to PEP 604, | will be used to designate union types from Python 3.10. So float | int will mean Union[float, int], i.e. a float or an int.
Question: This question is very similar to filtering np.nan values from pytorch in a -Dimensional tensor. The difference is that I want to apply the same concept to tensors of 2 or higher dimensions. I have a tensor that looks like this: import torch tensor = torch.Tensor( [[1, 1, 1, 1, 1], [float('nan'), float('nan'), float('nan'), float('nan'), float('nan')], [2, 2, 2, 2, 2]] )  >>> tensor.shape >>> [3, 5]  I would like to find the most pythonic / PyTorch way of to filter out (remove) the rows of the tensor which are nan. By filtering this tensor along the first (0th axis) I want to obtain a filtered_tensor which looks like this: >>> print(filtered_tensor) >>> torch.Tensor( [[1, 1, 1, 1, 1], [2, 2, 2, 2, 2]] ) >>> filtered_tensor.shape >>> [2, 5] 

Answer: Use PyTorch's isnan() together with any() to slice tensor's rows using the obtained boolean mask as follows: filtered_tensor = tensor[~torch.any(tensor.isnan(),dim=1)]  Note that this will drop any row that has a nan value in it. If you want to drop only rows where all values are nan replace torch.any with torch.all. For an N-dimensional tensor you could just flatten all the dims apart from the first dim and apply the same procedure as above: #Flatten: shape = tensor.shape tensor_reshaped = tensor.reshape(shape[0],-1) #Drop all rows containing any nan: tensor_reshaped = tensor_reshaped[~torch.any(tensor_reshaped.isnan(),dim=1)] #Reshape back: tensor = tensor_reshaped.reshape(tensor_reshaped.shape[0],*shape[1:])
Question: I am currently using jupyter notebook, and I want to delete variables that are used only within the cell, so that I won't accidentally misuse these variables in other cells. For example, I want to remove the variable myvar and loop variable i in the following codes: start = 1 stop = 2 for i in range(start, stop): pass myvar = "A" # other statements # ... del i, myvar # Runs ok, and variables i and myvar are deleted and I won't accidentally use i or myvar in another jupyter notebook cell  This works fine, but there are cases where some variables are actually not defined. In the following example, this throws an error since i is not defined because the loop is never run. The defined variables myvar is not deleted. start = 1 stop = 1 # Now we have stop equal to 1 for i in range(start, stop): pass myvar = "A" # other statements # ... del i, myvar # NameError, since i is not defined, and myvar is not deleted  I have used contextlib or try-except statements to avoid the error, but still myvar is not deleted. import contextlib start = 1 stop = 1 for i in range(start, stop): pass myvar = "A" # other statements # ... with contextlib.suppress(NameError): del i, myvar # Error suppressed, but myvar is not deleted  The only work-around is to wrap del statement for every variable I want to delete. import contextlib start = 1 stop = 1 for i in range(start, stop): pass myvar = "A" # other statements # ... with contextlib.suppress(NameError): del i with contextlib.suppress(NameError): del myvar # ... # It works but codes get messy if I need to delete many variables  However, this makes codes messy, especially if you have a lot of variables to delete (n lines if there are n variables). Is there any way to delete all defined and undefined variables safely with less messy codes, in one/two lines?

Answer: A one-liner to do del x safely: globals().pop('x', None);  There are many ways to do that, but they need more than 1 line of code, which I guess is not what you look for. Note ; at the end, which prevents the variable from being printed by Jupiter.
Question: What is the difference in Python between unpacking a function call with [], with () or with nothing? def f(): return 0, 1 a, b = f() # 1 [a, b] = f() # 2 (a, b) = f() # 3 

Answer: There is no difference. Regardless of what kind of syntactic sequence you use, the same byte code is generated. >>> def f(): ... return 0, 1 ... >>> import dis >>> dis.dis('[a,b] = f()') 1 0 LOAD_NAME 0 (f) 2 CALL_FUNCTION 0 4 UNPACK_SEQUENCE 2 6 STORE_NAME 1 (a) 8 STORE_NAME 2 (b) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE >>> dis.dis('(a,b) = f()') 1 0 LOAD_NAME 0 (f) 2 CALL_FUNCTION 0 4 UNPACK_SEQUENCE 2 6 STORE_NAME 1 (a) 8 STORE_NAME 2 (b) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE >>> dis.dis('a, b = f()') 1 0 LOAD_NAME 0 (f) 2 CALL_FUNCTION 0 4 UNPACK_SEQUENCE 2 6 STORE_NAME 1 (a) 8 STORE_NAME 2 (b) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE  In every case, you simply call f, then use UNPACK_SEQUENCE to produce the values to assign to a and b.  Even if you want to argue that byte code is an implementation detail of CPython, the definition of a chained assignment is not. Given x = [a, b] = f()  the meaning is the same as tmp = f() x = tmp [a, b] = tmp  x is assigned the result of f() (a tuple), not the "list" [a, b].  Finally, here is the grammar for an assignment statement: assignment_stmt ::= (target_list "=")+ (starred_expression | yield_expression) target_list ::= target ("," target)* [","] target ::= identifier | "(" [target_list] ")" | "[" [target_list] "]" | attributeref | subscription | slicing | "*" target  Arguably, the "[" [target_list] "]" could and should have been removed in Python 3. Such a breaking change would be difficult to implement now, given the stated preference to avoid any future changes to Python on the scale of the 2-to-3 transition.
Question: from sklearn import ensemble model = ensemble.RandomForestClassifier(n_estimators=10) model.fit(x,y) predictions = model.predict(new)  I know predict() uses predict_proba() to get the predictions, by computing the mean of the predicted class probabilities of the trees in the forest. I want to get the result of predict_proba() for the class predicted by the predict() method. What I'm doing is: first call predict() like in the above code, and for the probability I'm extracting the max probability from the trees like so: all_probabilities = model.predict_proba() class_probabilities = np.array([]) for tree in all_probabilities: class_probabilites = np.append(class_probabilities, tree.max())  Is this correct? If not, how can I extract the probability for the predicted class?

Answer: The predict_proba() method returns a two-dimensional array, containing the estimated probabilities for each instance and each class: import numpy as np from sklearn.ensemble import RandomForestClassifier X = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]) y = np.array([0, 0, 1, 1]) model = RandomForestClassifier() model.fit(X, y) model.predict_proba(X)  array([[0.91, 0.09], [0.91, 0.09], [0.25, 0.75], [0.05, 0.95]])  As you note, for each instance the predicted class is the class with the maximum probability. So one simple way to get the estimated probabilities for the predicted classes is to use np.max(): np.max(model.predict_proba(X), axis=1)  array([0.91, 0.91, 0.75, 0.95])
Question: I would like to use state-of-the-art LM T5 to get sentence embedding vector. I found this repository https://github.com/UKPLab/sentence-transformers As I know, in BERT I should take the first token as [CLS] token, and it will be the sentence embedding. In this repository I see the same behaviour on T5 model: cls_tokens = output_tokens[:, 0, :] # CLS token is first token  Does this behaviour correct? I have taken encoder from T5 and encoded two phrases with it: "I live in the kindergarden" "Yes, I live in the kindergarden"  The cosine similarity between them was only "0.2420". I just need to understand how sentence embedding works - should I train network to find similarity to reach correct results? Or I it is enough of base pretrained language model?

Answer: In order to obtain the sentence embedding from the T5, you need to take the take the last_hidden_state from the T5 encoder output: model.encoder(input_ids=s, attention_mask=attn, return_dict=True) pooled_sentence = output.last_hidden_state # shape is [batch_size, seq_len, hidden_size] # pooled_sentence will represent the embeddings for each word in the sentence # you need to sum/average the pooled_sentence pooled_sentence = torch.mean(pooled_sentence, dim=1)  You have now a sentence embeddings from T5
Question: I'm trying to install the pywin32 module on Ubuntu for python 3.6, I've tried pip3 install pywin32 and got the following output: Collecting pywin32 Could not find a version that satisfies the requirement pywin32 (from versions: ) No matching distribution found for pywin32  Then I tried pip3 install pypiwin32 and got the following output: Collecting pypiwin32 Using cached https://files.pythonhosted.org/packages/d0/1b/2f292bbd742e369a100c91faa0483172cd91a1a422a6692055ac920946c5/pypiwin32-223-py3-none-any.whl Collecting pywin32>=223 (from pypiwin32) Could not find a version that satisfies the requirement pywin32>=223 (from pypiwin32) (from versions: ) No matching distribution found for pywin32>=223 (from pypiwin32) 

Answer: The pywin32 and pypiwin32 is "Python extensions for Microsoft Windows Provides access to much of the Win32 API, the ability to create and use COM objects, and the Pythonwin environment." One supported OS is Microsoft Windows, because you can access the Win32 API only from Windows.  Source: https://pypi.org/project/pywin32/
Question: I need to implement custom exceptions for handling gRPC request errors with Python. For HTTP requests it's straightforward - requests library catches it well when there's an error code etc. I am looking for analogous ways for gRPC to do something like: try: # send gRPC request except SomeGRPCException as e: # custom handle  Is there a way to handle gRPC errors like that in Python? Or with gRPC it wouldn't work like in the example?

Answer: For simple RPC error handling, you can use try-catch: try: response = stub.SayHello(...) except grpc.RpcError as rpc_error: if rpc_error.code() == grpc.StatusCode.CANCELLED: pass elif rpc_error.code() == grpc.StatusCode.UNAVAILABLE: pass else: print(f"Received unknown RPC error: code={rpc_error.code()} message={rpc_error.details()}")  For complex RPC error handling, you may need to utilize the trailing metadata: https://github.com/grpc/grpc/tree/master/examples/python/errors In most cases, try-catch should be sufficient.
Question: How can I use TF-IDF vectorizer from the scikit-learn library to extract unigrams and bigrams of tweets? I want to train a classifier with the output. This is the code from scikit-learn: from sklearn.feature_extraction.text import TfidfVectorizer corpus = [ 'This is the first document.', 'This document is the second document.', 'And this is the third one.', 'Is this the first document?', ] vectorizer = TfidfVectorizer() X = vectorizer.fit_transform(corpus) 

Answer: TfidfVectorizer has an ngram_range parameter to determin the range of n-grams you want in the final matrix as new features. In your case, you want (1,2) to go from unigrams to bigrams: vectorizer = TfidfVectorizer(ngram_range=(1,2)) X = vectorizer.fit_transform(corpus).todense() pd.DataFrame(X, columns=vectorizer.get_feature_names()) and and this document document is first first document \ 0 0.000000 0.000000 0.314532 0.000000 0.388510 0.388510 1 0.000000 0.000000 0.455513 0.356824 0.000000 0.000000 2 0.357007 0.357007 0.000000 0.000000 0.000000 0.000000 3 0.000000 0.000000 0.282940 0.000000 0.349487 0.349487 is is the is this one ... the the first \ 0 0.257151 0.314532 0.000000 0.000000 ... 0.257151 0.388510 1 0.186206 0.227756 0.000000 0.000000 ... 0.186206 0.000000 2 0.186301 0.227873 0.000000 0.357007 ... 0.186301 0.000000 3 0.231322 0.000000 0.443279 0.000000 ... 0.231322 0.349487 ...
Question: I have a function which validates its argument to accept only values from a given list of valid options. Typing-wise, I reflect this behavior using a Literal type alias, like so: from typing import Literal VALID_ARGUMENTS = ['foo', 'bar'] Argument = Literal['foo', 'bar'] def func(argument: 'Argument') -> None: if argument not in VALID_ARGUMENTS: raise ValueError( f'argument must be one of {VALID_ARGUMENTS}' ) # ...  This is a violation of the DRY principle, because I have to rewrite the list of valid arguments in the definition of my Literal type, even if it is already stored in the variable VALID_ARGUMENTS. How can I create the Argument Literal type dynamically, given the VALID_ARGUMENTS variable? The following things do not work: from typing import Literal, Union, NewType Argument = Literal[*VALID_ARGUMENTS] # SyntaxError: invalid syntax Argument = Literal[VALID_ARGUMENTS] # Parameters to generic types must be types Argument = Literal[Union[VALID_ARGUMENTS]] # TypeError: Union[arg, ...]: each arg must be a type. Got ['foo', 'bar']. Argument = NewType( 'Argument', Union[ Literal[valid_argument] for valid_argument in VALID_ARGUMENTS ] ) # Expected type 'Type[_T]', got 'list' instead  Can it be done at all?

Answer: Go the other way around, and build VALID_ARGUMENTS from Argument: Argument = typing.Literal['foo', 'bar'] VALID_ARGUMENTS: typing.Tuple[Argument, ...] = typing.get_args(Argument)  I've used a tuple for VALID_ARGUMENTS here, but if for some reason you really prefer a list, you can get one: VALID_ARGUMENTS: typing.List[Argument] = list(typing.get_args(Argument))   It's possible at runtime to build Argument from VALID_ARGUMENTS, but doing so is incompatible with static analysis, which is the primary use case of type annotations. Doing so is also considered semantically invalid - the spec forbids parameterizing Literal with dynamically computed parameters. The runtime implementation simply doesn't have the information it would need to validate this. Building VALID_ARGUMENTS from Argument is the way to go.
Question: I have dataframe tha tlooks similar to this: >>>Hour Level value 0 7 H 1.435 1 7 M 3.124 2 7 L 5.578 3 8 H 0.435 4 8 M 2.124 5 8 L 4.578  I want to create line chart in plotly that will have different line style based in the column "level". Right now I have the line chart with the deafult line style: import plotly.graph_objects as go fig = px.line(group, x="Hour", y="value",color='level', title='Graph',category_orders={'level':['H','M','L']} ,color_discrete_map={'H':'royalblue','M':'orange','L':'firebrick'}) fig.show()   I would like to control the linestyle for each level. until know I saw that the only way to do this is to add for each "level" but using add_trace as following: # Create and style traces fig.add_trace(go.Scatter(x="Hour", y="value", name='H', line=dict(dash='dash'))) fig.add_trace(go.Scatter(x="Hour", y="value", name = 'M', line=dict(dash='dot'))) fig.show()  but I keep getting this error:  ValueError: Invalid value of type 'builtins.str' received for the 'x' property of scatter Received value: 'Hour' The 'x' property is an array that may be specified as a tuple, list, numpy array, or pandas Series   My end goal is to control the linestyle of the lines in my charts, better if I can do thatinside the part of "px.line"

Answer: One way you can set different styles through variables in your dataframe is: line_dash='Level'  Plot  Complete code import plotly.graph_objects as go import pandas as pd import numpy as np import plotly.io as pio import plotly.express as px group = pd.DataFrame({'Hour': {0: 7, 1: 7, 2: 7, 3: 8, 4: 8, 5: 8}, 'Level': {0: 'H', 1: 'M', 2: 'L', 3: 'H', 4: 'M', 5: 'L'}, 'value': {0: 1.435, 1: 3.1239999999999997, 2: 5.577999999999999, 3: 0.435, 4: 2.124, 5: 4.578}}) import plotly.graph_objects as go fig = px.line( group, x="Hour", y="value", color='Level', title='Graph', category_orders={'Level':['H','M','L']}, color_discrete_map={'H':'royalblue','M':'orange','L':'firebrick'}, line_dash='Level' ) fig.show()
Question: I just started using the 'Black' formatter module with Visual Studio Code. Everything was going well till I just noticed that it uses double quotes over single quotes which I already was using in my code... And it overrode that... So, is there an Black argument that I could add to Visual Studio Code which solves this problem?

Answer: You can use the --skip-string-normalization option at the command line, or in your Visual Studio Code options. See The Black code style, Strings. For example: { ... "python.formatting.provider": "black", "python.formatting.blackArgs": [ "--skip-string-normalization", "--line-length", "100" ] ... }
Question: I've tried to search this a number of times but I don't see it answered so here goes... I often use pandas to clean up a dataframe and conform it to my needs. With this comes a lot of .loc accessing to query it and return values. Depending on what I am doing (and column lengths), this can get pretty lengthy. Given PEP8 constrains to 79 characters a line, are there any best practices? Some examples below (these are simplified and for explanatory purposes): missing_address_df = address_df.loc[address_df['address'].notnull()].copy()  or multiple query points: nc_drive_df = address.loc[(address_df['address'].str.contains('drive')) & (address_df['state'] == 'NC')] 

Answer: I'd advise two things  Ignore PEP 8's 80 char advice, but try to keep to 120 or 150 lines Keeping some line length requirement makes sense to aid readability, but if you're trying to keep to 80 chars in (for example) a class method, it will lead to worse and less-readable code PEP 8 actually has a section on this, A Foolish Consistency is the Hobgoblin of Little Minds, which describes cases you should deviate from its other advice, for example   When applying the guideline would make the code less readable, even for someone who is used to reading code that follows this PEP    split the .loc contents onto multiple lines nc_drive_df = address.loc[ (address_df['address'].str.contains('drive')) & \ (address_df['state'] == 'NC') ]    It's hard to be objective about when code "looks bad", despite being valid syntax, but you will experience it. Practically, PEP 8 and Cyclomatic Complexity checkers are tools which will help you fight against and defend and propose code styles in a scientific way.  If you have a great many boolean statements, you (often must) break them up with parentheses to clarify their order nc_drive_df = address.loc[ ( (address_df['address'].str.contains('drive')) & \ (address_df['state'] == 'NC') ) || ( address_df['zip'] == "00000" ) ]  This is somewhat in conflict with conventional Python operators, which are suggested to preceed lines (PEP8), but I challenge this when forming a boolean array because the member Series must be the same shape to get a good result and it's generally easier to observe and reason about them when working with several DataFrames when they're visually aligned. Finally, often when doing scientific Python, you should absolutely try many possibilities against a partial and full data if possible to draw good performance conclusions, consider their readability to be second, and provide excellent comments about and linking to your research, etc. over any particular style.
Question:  In using pandas, how can I display a table similar to this one. I think I have to use a dataframe similar to df = pandas.DataFrame(results) and display it with display.display(df) but from there I don't know what to do?

Answer: You can pass in a dictionary as data when you use pd.DataFrame: >>> import pandas as pd >>> d = { ... 'Algothime': ['KNN', 'SVM', 'MLP'], ... 'Param. 1': ['-', '-', '-'], ... 'Param. 2': ['-', '-', '-'], ... 'Plage param. 1': ['-', '-', '-'], ... 'Plage param. 2': ['-', '-', '-'], ... } >>> df = pd.DataFrame(data=d) >>> df Algothime Param. 1 Param. 2 Plage param. 1 Plage param. 2 0 KNN - - - - 1 SVM - - - - 2 MLP - - - -  If you want that specific style you can use Google Colab or something similar if you don't have Jupyter installed locally: