eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: i have df below   Cost,Reve 0,3 4,0 0,0 10,10 4,8 len(df['Cost']) = 300 len(df['Reve']) = 300  I need to divide df['Cost'] / df['Reve'] Below is my code df[['Cost','Reve']] = df[['Cost','Reve']].apply(pd.to_numeric)  I got the error ValueError: Columns must be same length as key df['C/R'] = df[['Cost']].div(df['Reve'].values, axis=0)  I got the error ValueError: Wrong number of items passed 2, placement implies 1

Answer: Problem is duplicated columns names, verify: #generate duplicates df = pd.concat([df, df], axis=1) print (df) Cost Reve Cost Reve 0 0 3 0 3 1 4 0 4 0 2 0 0 0 0 3 10 10 10 10 4 4 8 4 8 df[['Cost','Reve']] = df[['Cost','Reve']].apply(pd.to_numeric) print (df) # ValueError: Columns must be same length as key  You can find this columns names: print (df.columns[df.columns.duplicated(keep=False)]) Index(['Cost', 'Reve', 'Cost', 'Reve'], dtype='object')  If same values in columns is possible remove duplicated by: df = df.loc[:, ~df.columns.duplicated()] df[['Cost','Reve']] = df[['Cost','Reve']].apply(pd.to_numeric) #simplify division df['C/R'] = df['Cost'].div(df['Reve']) print (df) Cost Reve C/R 0 0 3 0.0 1 4 0 inf 2 0 0 NaN 3 10 10 1.0 4 4 8 0.5
Question: This seems to be one of the most common questions about LSTMs in PyTorch, but I am still unable to figure out what should be the input shape to PyTorch LSTM. Even after following several posts (1, 2, 3) and trying out the solutions, it doesn't seem to work.  Background: I have encoded text sequences (variable length) in a batch of size 12 and the sequences are padded and packed using pad_packed_sequence functionality. MAX_LEN for each sequence is 384 and each token (or word) in the sequence has a dimension of 768. Hence my batch tensor could have one of the following shapes: [12, 384, 768] or [384, 12, 768]. The batch will be my input to the PyTorch rnn module (lstm here). According to the PyTorch documentation for LSTMs, its input dimensions are (seq_len, batch, input_size) which I understand as following. seq_len - the number of time steps in each input stream (feature vector length). batch - the size of each batch of input sequences. input_size - the dimension for each input token or time step. lstm = nn.LSTM(input_size=?, hidden_size=?, batch_first=True) What should be the exact input_size and hidden_size values here?

Answer: You have explained the structure of your input, but you haven't made the connection between your input dimensions and the LSTM's expected input dimensions. Let's break down your input (assigning names to the dimensions):  batch_size: 12 seq_len: 384 input_size / num_features: 768  That means the input_size of the LSTM needs to be 768. The hidden_size is not dependent on your input, but rather how many features the LSTM should create, which is then used for the hidden state as well as the output, since that is the last hidden state. You have to decide how many features you want to use for the LSTM. Finally, for the input shape, setting batch_first=True requires the input to have the shape [batch_size, seq_len, input_size], in your case that would be [12, 384, 768]. import torch import torch.nn as nn # Size: [batch_size, seq_len, input_size] input = torch.randn(12, 384, 768) lstm = nn.LSTM(input_size=768, hidden_size=512, batch_first=True) output, _ = lstm(input) output.size() # => torch.Size([12, 384, 512])
Question: I have a list of integers, and I need to split it into a given number of sublists (with no restrictions on order or the number of elements in each), in a way that minimizes the average difference in sums of each sublist. For example: >>> x = [4, 9, 1, 5] >>> sublist_creator(x, 2) [[9], [4, 1, 5]]  because list(map(sum, sublist_creator(x, 2))) yields [9, 10], minimizing the average distance. Alternatively, [[9, 1], [4, 5]] would have been equally correct, and my use case has no preference between two possibilities. The only way I can think of to do this is by checking, iteratively, all possible combinations, but I'm working with a list of ~5000 elements and need to split it into ~30 sublists, so that approach is prohibitively expensive.

Answer: Here's the outline:  create N empty lists sort() your input array in ascending order pop() the last element from the sorted array append() the popped element to the list with the lowest sum() of the elements repeat 3 and 4 until input array is empty profit!!!  With M=5000 elements and N=30 lists this approach might take about O(N*M) if you carefully store the intermediate sums of the sublists instead of calculating them from the scratch every time.
Question: I am trying to display the contents of an Excel file in a Jupyter Notebook. However, a column named definition of the Excel sheet contains long strings. So when I display the DataFrame in the notebook, the long strings are truncated by ellipsis (...).  Is there a way to display the complete contents of the column in Jupyter Notebook? Since there is clearly space to the right that can utilized by the Definition column.

Answer: You can use options.display.max_colwidth to specify you want to see more in the default representation: In [2]: df Out[2]: one 0 one 1 two 2 This is very long string very long string very... In [3]: pd.options.display.max_colwidth Out[3]: 50 In [4]: pd.options.display.max_colwidth = 100 In [5]: df Out[5]: one 0 one 1 two 2 This is very long string very long string very long string veryvery long string  reference - Print very long string completely in pandas dataframe
Question: I've recently upgraded Python to 3.7.6 and my existing code:  df['Simple_Avg_Return'] = df.groupby(['YF_Ticker'])['Share_Price_Delta_Percent', 'Dividend_Percent'].transform( sum).divide(2).round(2)  Is now throwing this warning: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.  How would I go about converting this to a list as advised and where?

Answer: You need to use an extra bracket around ['Share_Price_Delta_Percent', 'Dividend_Percent'] Like this, df['Simple_Avg_Return'] = df.groupby(['YF_Ticker'])[['Share_Price_Delta_Percent', 'Dividend_Percent']].transform( sum).divide(2).round(2)  Quoting @ALollz comment  The decision was made https://github.com/pandas-dev/pandas/issues/23566. To keep compatibility between 0.25 and 1.0 they didn't remove the feature but added a warning in 1.0. Likely it will be removed in the next major deprecation cycle.  Source
Question: I'm referring to the documentation available here to understand how the return types are defined while using function annotations. I couldn't understand what the str in Dict[str, Any] refers to. Does str refer to the dictionary's keys and Any (meaning, it can be a string or an int) refer to the type of the dictionary's value? EDIT: In the above-mentioned link, it is mentioned  The PEP 484 type Dict[str, Any] would be suitable, but it is too lenient, as arbitrary string keys can be used, and arbitrary values are valid.  Could someone explain what does arbitrary string keys refer to? I understand keys are strings, but when we say arbitrary string keys do we just mean that the dictionary can take any key that is a string? Or does the word arbitrary hold any other significance here?

Answer: Yep! Normally python variables are mutable (types can change), but specifying it like this is good documentation and makes it very clear what goes where.  More usage documentation can be found here! https://docs.python.org/3/library/typing.html EDIT: Elaborate on answer on updated question The PEP documentation you refer to identifies how type-hints, albeit useful, are still error prone.  By specifying Dict[str, Any], the str can be "arbitrary", meaning it can be a string key of anything (ie. name, age, height, humidity). Out of these keys, one may consider "humidity" shouldn't be one of the string-keys for this dict, but there isn't a way to check, or enforce that until some error occurs down the road (hence, it is "arbitrary"; no rules to govern what are "allowed" string-keys of this dict). Therefore, per the documentation, specifying this class (derived from TypedDict):  class Movie(TypedDict): name: str year: int   will specifically limit users to create a new kind of dict (a TypedDict) with string-keys name and year (with respective values typed str and int). Users will not be able to "arbitrarily" add a new key (ie. humidity) to this "Movie" TypedDict, or otherwise assign a non-int value to Movie["year"].
Question: I'm trying to run a python file in VSCode using python3. I can run using integrated terminal like it says in the microsoft vscode tutorial on python. However, I would like the program to print in the output tab and not take up the terminal window. However I'm getting this error.  The standard code runner config file launch.json, looks like this; "version": "0.2.0", "configurations": [ { "name": "Python: Current File", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal" } ]  I've tried to set my python path in VSCode in settings.json ... "python.pythonPath": "python3", "code-runner.executorMap": { "python3": "/usr/bin/python3" }  I've also set an alias for python -> python3 (as my ubuntu 20.04 doesn't come with python2 anymore) alias python="python3"  However, I keep getting the above error. Any Ideas?

Answer: Solution in 2024 Step1 : Goto settings of Code Runner extension Step2 : Find the section  Code-runner: Executor Map  And click on  Edit in settings.json  Step3 : Now change the setting for python Before "code-runner.executorMap": { ... "python": "python -u" ... }  Change this to "code-runner.executorMap": { ... "python": "python3 -u" ... }
Question: I have an issue with tf.callbacks.ModelChekpoint. As you can see in my log file, the warning comes always before the last iteration where the val_acc is calculated. Therefore, Modelcheckpoint never finds the val_acc Epoch 1/30 1/8 [==>...........................] - ETA: 19s - loss: 1.4174 - accuracy: 0.3000 2/8 [======>.......................] - ETA: 8s - loss: 1.3363 - accuracy: 0.3500 3/8 [==========>...................] - ETA: 4s - loss: 1.3994 - accuracy: 0.2667 4/8 [==============>...............] - ETA: 3s - loss: 1.3527 - accuracy: 0.3250 6/8 [=====================>........] - ETA: 1s - loss: 1.3042 - accuracy: 0.3333 WARNING:tensorflow:Can save best model only with val_acc available, skipping. 8/8 [==============================] - 4s 482ms/step - loss: 1.2846 - accuracy: 0.3375 - val_loss: 1.3512 - val_accuracy: 0.5000 Epoch 2/30 1/8 [==>...........................] - ETA: 0s - loss: 1.0098 - accuracy: 0.5000 3/8 [==========>...................] - ETA: 0s - loss: 0.8916 - accuracy: 0.5333 5/8 [=================>............] - ETA: 0s - loss: 0.9533 - accuracy: 0.5600 6/8 [=====================>........] - ETA: 0s - loss: 0.9523 - accuracy: 0.5667 7/8 [=========================>....] - ETA: 0s - loss: 0.9377 - accuracy: 0.5714 WARNING:tensorflow:Can save best model only with val_acc available, skipping. 8/8 [==============================] - 1s 98ms/step - loss: 0.9229 - accuracy: 0.5750 - val_loss: 1.2507 - val_accuracy: 0.5000  This is my code for training the CNN. callbacks = [ TensorBoard(log_dir=r'C:\Users\reda\Desktop\logs\{}'.format(Name), histogram_freq=1), ModelCheckpoint(filepath=r"C:\Users\reda\Desktop\checkpoints\{}".format(Name), monitor='val_acc', verbose=2, save_best_only=True, mode='max')] history = model.fit_generator( train_data_gen, steps_per_epoch=total_train // batch_size, epochs=epochs, validation_data=val_data_gen, validation_steps=total_val // batch_size, callbacks=callbacks) 

Answer: I know how frustrating these things can be sometimes..but tensorflow requires that you explicitly write out the name of metric you are wanting to calculate You will need to actually say 'val_accuracy' metric = 'val_accuracy' ModelCheckpoint(filepath=r"C:\Users\reda.elhail\Desktop\checkpoints\{}".format(Name), monitor=metric, verbose=2, save_best_only=True, mode='max')]  Hope this helps =) *** As later noted by BlueTurtle (Give their answer a thumbs up please, likely still beneath this) you also need to use the full metric name to match your model.compile, ModelCheckpoint, and EarlyStopping.
Question: Documentation for str.swapcase() method says:  Return a copy of the string with uppercase characters converted to lowercase and vice versa. Note that it is not necessarily true that s.swapcase().swapcase() == s.  I can't think of an example where s.swapcase().swapcase() != s, can anyone think of one?

Answer: A simple example would be: s = "ß" print(s.swapcase().swapcase())   Ouput: ss   ß is German lowercase double s (The "correct" uppercase version would be ẞ). The reason this happens is that the Unicode standard has defined the capitalization of ß to be SS:  The data in this file, combined with # the simple case mappings in UnicodeData.txt, defines the full case mappings # Lowercase_Mapping (lc), Titlecase_Mapping (tc), and Uppercase_Mapping (uc). ... # The entries in this file are in the following machine-readable format: # # <code>; <lower>; <title>; <upper>; (<condition_list>;)? # <comment> ... # The German es-zed is special--the normal mapping is to SS. # Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase(<es-zed>)) 00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S   (00DF is ß, 0053 is S, and 0073 is s)
Question: I'm currently researching Black as our default formatter, but, I'm having some edge cases that don't format well and I want to know if there's a way to get the result I want. Black's documentation partially explores my problem, I have a dictionary expression spread horizontally, and I want to keep it that way since I'm expecting lines to be added, e.g.: # Black would keep this as-is because of the trailing comma TRANSLATIONS = { "en_us": "English (US)", "pl_pl": "polski", }  But in my case the dictionary is inside a list comprehension: res = [ { 'id': item.id, 'name': item.name, } for item in items.select() ]  Which Black collapses, regardless of the trailing comma, like so: res = [ {"id": item.id, "name": item.name,} for item in items.select() ]  Is there a way of telling Black to retain the horizontal structure in these cases?

Answer: It seems that black addressed this issue. At the time of writing this answer, using black version 20.8b1, the formatting is done as I was hoping for. As long as there is a magic trailing comma on the last item in the dictionary expression, black will format the code within the list comprehension. res = [ { "id": item.id, "name": item.name, } for item in items.select() ]  Will format to: res = [ { "id": item.id, "name": item.name, } for item in items.select() ]
Question: Python doc __future__ In the python docs about __future__ there is a table where it shows that annotations are "optional in" 3.7.0b1 and "mandatory in" 4.0 but I am still able to use annotations in 3.8.2 without importing annotations. Given that, what is the use of it? >>> def add_int(a:int, b:int) -> int: ... return a + b >>> add_int.__annotations__ {'a': <class 'int'>, 'b': <class 'int'>, 'return': <class 'int'>}  I doubt I clearly understand the meaning of "optional in" and "mandatory in" here

Answer: Mandatory is an interesting word choice. I guess it means that it's by default in the language. You don't have to enable it with from __future__ import annotations The annotations feature are referring to the PEP 563: Postponed evaluation of annotations. It's an enhancement to the existing annotations feature which was initially introduced in python 3.0 and redefined as type hints in python 3.5, that's why your code works under python 3.8. Here's what optional from __future__ import annotations changes in python 3.7+: class A: def f(self) -> A: # NameError: name 'A' is not defined pass  but this works from __future__ import annotations class A: def f(self) -> A: pass  See this chapter in python 3.7 what's new about postponed annotations:  Since this change breaks compatibility, the new behavior needs to be enabled on a per-module basis in Python 3.7 using a __future__ import: from __future__ import annotations It will become the default in Python 3.10*.  * it was announced to be default in 3.10 (when python3.7 was released), but it was now moved to a later release
Question: I'm using protoBuffer 3 in python it is not showing bool fields correctly.  entity_proto message Foo { string id = 1; bool active = 3; }  Setting values in python.  foo = entity_proto.Foo(id='id-123', active=True) print(foo) # id: id-123 # active: True # But if you set the value False it does not show 'active' in print statement foo = entity_proto.Foo(id='id-123', active=False) print(foo) # id: id-123  if you try to print print(foo.active) output is False which is somehow OK. Main problem is come when I use Http trancoding if I try to print console.log(foo.active) is give me undefined not false (lang: JavaScript)  Can someone please let me know why it is not showing for False values. 

Answer: Protobuf has default values for fields, such as boolean false, numeric zero, or the empty string. It doesn't bother encoding those since it's a waste of space and/or bandwidth (when transmitting). That's probably why it's not showing up. A good way to check this would be to set id to an empty string and see if it behaves similarly: foo = entity_proto.Foo(id='', active=True) print(foo) # active: True (I suspect).  The solution depends really on where that undefined is coming from. Either Javascript has a real undefined value, in which case you can use the null/undefined coalescing operator: console.log(foo.active ?? false)  Or, if this HTTP transcoder is doing something like creating an literal "undefined" string, you'll have to figure out how to turn (what is probably) None into "false".
Question: What is the difference between the Model.clean() method and the Model.clean_fields() method?. When should I use one or the other?.  According to the Model.clean_fields() method documentation:  This method will validate all fields on your model. The optional exclude argument lets you provide a list of field names to exclude from validation. It will raise a ValidationError if any fields fail validation. . . .  So the Model.clean_fields() method is used to validate the fields of my model.  According to the Model.clean() method documentation:  This method should be used to provide custom model validation, and to modify attributes on your model if desired. For instance, you could use it to automatically provide a value for a field, or to do validation that requires access to more than a single field . . .  But, this method should be used to make other validations, but among those, you can perform validations to different fields. And in the Django examples of the Model.clean() method, validations are made to fields: class Article(models.Model): ... def clean(self): # Don't allow draft entries to have a pub_date. if self.status == 'draft' and self.pub_date is not None: raise ValidationError(_('Draft entries may not have a publication date.')) # Set the pub_date for published items if it hasn't been set already. if self.status == 'published' and self.pub_date is None: self.pub_date = datetime.date.today()   So for example, if I want to validate a field, in which of the two methods should I do it? And for what exactly should the methods be used? since I don't understand very well why they should be used exactly. It would be very helpful to provide examples where you can see the difference between the methods, to know exactly what the methods should be used for.

Answer:  if I want to validate a field, in which of the two methods should I do it?  Then you add that normally as validator [Django-doc]: from django.core.exceptions import ValidationError def even_validator(value): if value % 2: ValidationError('The value should be even') class MyModel(models.Model): my_fields = models.IntegerField(validators=[even_validator]) The .clean_fields() method [Django-doc] will then call the cleaning methods and validation methods of the individual fields, and you can pass parameters to specify if you want to exclude validating a certain field. But normally you do not override .clean_fields() itself. The .clean() method [Django-doc] on the other hand is a validation method that is used if multiple fields are involved. You can override this, although in that case you normally also call the super().clean() method to do validations defined at the base class.
Question: I am curious what would be the best way to specify that a type is not just a int but a positive int in Python. Examples: # As function argument def my_age(age: int) -> str: return f"You are {age} years old." # As class property class User: id: int  In both of these situations a negative value would be erroneous. It would be nice to be warned by my IDE/linter. Is there a simple way to specify an integer as positive using type-hints?

Answer: Little late to the party but there is now a library called annotated-types that provides exactly what you want and is the 'official' way to go about this, per PEP-593. So for your example of positive integers, you could make use of the built-in typing.Annotated and annotated-types' Gt predicate, like this: from typing import Annotated from annotated_types import Gt def my_age(age: Annotated[int, Gt(0)]) -> str: return f"You are {age} years old." # As class property class User: id: Annotated[int, Gt(0)]  Note that like all things related to type hints in Python, this doesn't actually enforce the constraint at runtime. If you want to enforce it you would still have to build assertions yourself.
Question: The trouble is this.  Lets say we have a pandas df that can be generated using the following: month=['dec','dec','dec','jan','feb','feb','mar','mar'] category =['a','a','b','b','a','b','b','b'] sales=[1,10,2,5,12,4,3,1] df = pd.DataFrame(list(zip(month,category,sales)), columns =['month', 'cat','sales']) print(df) | month cat sales | |--------------------| | 0 dec a 1 | | 1 dec a 10 | | 2 dec b 2 | | 3 jan b 5 | | 4 feb a 12 | | 5 feb b 4 | | 6 mar b 3 | | 7 mar b 1 |  then let's suppose we would like a count of each category by month.  so we go and do something like  df=df.groupby(['month','cat']).sales.sum().reset_index() print(df) | month cat sales | |--------------------| | 0 dec a 11 | | 1 dec b 2 | | 2 feb a 12 | | 3 feb b 4 | | 4 jan b 5 | | 5 mar b 4 |  But what we'd like to see is:  | month cat sales | |--------------------| | 0 dec a 11 | | 1 dec b 2 | | 2 feb a 12 | | 3 feb b 4 | | 4 jan b 5 | | 5 jan a 0 | | 6 mar b 4 | | 7 mar a 0 |  Where the difference is categories that did not show up in a particular month would still show up just with zero as their total.  It's probable this has been asked before, but I couldn't find it. If you point me in the direction of the question, we'll go ahead and delete this one. 

Answer: Continuing from where you stopped, a combo of stack and unstack will give you your required output: res = ( df.groupby(['month', 'cat']) .sales.sum() .unstack(fill_value=0) # Unstack and fill value for the null column .stack() # Return to groupby form and reset .reset_index(name='sales') )  The output of res: >>> res month cat sales 0 dec a 11 1 dec b 2 2 feb a 12 3 feb b 4 4 jan a 0 5 jan b 5 6 mar a 0 7 mar b 4  You can also work with categoricals and set observed to False; this will ensure that all possible combinations are presented in the final output. (df.astype({'month' : 'category', 'cat' : 'category'}) .groupby(['month', 'cat'], as_index = False, observed = False) .sum(numeric_only = True) ) month cat sales 0 dec a 11 1 dec b 2 2 feb a 12 3 feb b 4 4 jan a 0 5 jan b 5 6 mar a 0 7 mar b 4