eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So I have a df like this In [1]:data= {'Group': ['A','A','A','A','A','A','B','B','B','B'], 'Name': [ ' Sheldon Webb',' Traci Dean',' Chad Webster',' Ora Harmon',' Elijah Mendoza',' June Strickland',' Beth Vasquez',' Betty Sutton',' Joel Gill',' Vernon Stone'], 'Performance':[33,64,142,116,122,68,95,127,132,80]} In [2]:df = pd.DataFrame(data, columns = ['Group', 'Name','Performance']) Out[1]: Group Name Performance 0 A Sheldon Webb 33 1 A Traci Dean 64 2 A Chad Webster 142 3 A Ora Harmon 116 4 A Elijah Mendoza 122 5 A June Strickland 68 6 B Beth Vasquez 95 7 B Betty Sutton 127 8 B Joel Gill 132 9 B Vernon Stone 80  I want to sort it in such an alternating way that within a group, say group "A", the first row should have its highest performing person (in this case "Chad Webster") and then in the second row the least performing (which is "Sheldon Webb"). The output I am looking for would look something like this: Out[2]: Group Name Performance 0 A Chad Webster 142 1 A Sheldon Webb 33 2 A Elijah Mendoza 122 3 A Traci Dean 64 4 A Ora Harmon 116 5 A June Strickland 68 6 B Joel Gill 132 7 B Vernon Stone 80 8 B Betty Sutton 127 9 B Beth Vasquez 95  You can see the sequence is alternating between the highest and lowest within a group.

Answer: Take the sorted order and then apply a quadratic function to it where the root is 1/2 the length of the array (plus some small offset). This way the highest rank is given to the extremal values (the sign of the eps offset determines whether you want a the highest value ranked above the lowest value). I added a small group at the end to show how it properly handles repeated values or an odd group size. def extremal_rank(s): eps = 10**-4 y = (pd.Series(np.arange(1, len(s)+1), index=s.sort_values().index) - (len(s)+1)/2 + eps)**2 return y.reindex_like(s) df['rnk'] = df.groupby('Group')['Performance'].apply(extremal_rank) df = df.sort_values(['Group', 'rnk'], ascending=[True, False])    Group Name Performance rnk 2 A Chad Webster 142 6.2505 0 A Sheldon Webb 33 6.2495 4 A Elijah Mendoza 122 2.2503 1 A Traci Dean 64 2.2497 3 A Ora Harmon 116 0.2501 5 A June Strickland 68 0.2499 8 B Joel Gill 132 2.2503 9 B Vernon Stone 80 2.2497 7 B Betty Sutton 127 0.2501 6 B Beth Vasquez 95 0.2499 11 C b 110 9.0006 12 C c 68 8.9994 10 C a 110 4.0004 13 C d 68 3.9996 15 C f 70 1.0002 16 C g 70 0.9998 14 C e 70 0.0000
Question: I am learning about file objects in python but whenever i try to open file it shows the following error. I have already checked that file is in same directory and it exists this error occurs only if i name my file as test if i use any other name then it works fine here's my CODE f = open('C:\\Users\Tanishq\Desktop\python tutorials\test.txt', 'r')  here's the ERROR  Traceback (most recent call last): File "C:/Users/Tanishq/Desktop/question.py", line 1, in <module> f = open('C:\\Users\Tanishq\Desktop\python tutorials\test.txt', 'r') OSError: [Errno 22] Invalid argument: 'C:\\Users\\Tanishq\\Desktop\\python tutorials\test.txt' 

Answer: Your issue is with backslashing characters like \T : Try: f = open(r'C:\\Users\Tanishq\Desktop\python tutorials\test.txt', 'r')  Python uses \ to denote special characters. Therefore, the string you provided does not actually truly represent the correct filepath, since Python will interpret \Tanishq\ differently than the raw string itself. This is we we place the r in front of it. This lets Python know that we do indeed want to use the raw string and to treat \ as a normal character.
Question: I found strange behavior with Python's in operator d = {} 'k' in d == False # False!  I thought it's because of precedence: ('k' in d) == False # True, it's okay 'k' in (d == False) # Error, it's also okay  But, what precedence evaluates the following expression then? d = {} 'k' in d == False  If it's because of wrong precedence why it doesn't fire an error like if: 'k' in (d == False)  In other words, what happens under the hood of Python with this expression? 'k' in d == False 

Answer: in is considered a comparison operator, and so it is subject to comparison chaining. 'k' in d == False  is equivalent to 'k' in d and d == False  because both in and == are comparison operators. You virtually never need direct comparison to Boolean literals, though. The "correct" expression here is 'k' not in d.  For reference, this is described in the Python documentation, under 6.10. Comparisons:  comparison ::= or_expr (comp_operator or_expr)* comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!=" | "is" ["not"] | ["not"] "in"   and  Comparisons can be chained arbitrarily, e.g., x < y <= z is equivalent to x < y and y <= z, except that y is evaluated only once (but in both cases z is not evaluated at all when x < y is found to be false).
Question: I am trying to understand a piece of Python code. First, I have the following file structure: folder1-- model __init__.py file1.py file2.py __init__.py  The __init__.py in the folder folder1 has nothing. The __init__.py in the folder model has the following: import os files = os.listdir(os.path.split(os.path.realpath(__file__))[0]) files.remove('__init__.py') for file in files: if file.endswith('.py'): exec('from .{} import *'.format(file[:-3]))  With that said, I have some code in python that uses all the above Now, I am trying to understand the following code from folder1 import model as mymodel  My first question is what does this do? I mean model is a folder name right? it is not an object. or is it? What is exactly importing as mymodel here? then later in the same code it says global args, cfg, is_fov120 args = parser.parse_args() model = getattr(mymodel, args.arch)(sync_bn=False)  Apparently there is some arguments called arch. What is happening here and what does model have after this? Edit When I do print(mymodel) I get <module 'folder1.model' from 'C:\\path\\to\\folder1\\model\\__init__.py'> Investigating even more I can see that I have imported all the objects from the files in the folder model. mymodel.files gives the list of files in the folder, and I can call mymodel.somevariable if some variable was defined in file1.py or file2.py. As for classes I have to create an object first like x=mymodel.aClass() and then I can access the elements of the object x.someElement. Finally I found out that getattr is getting a class from the files inside model and I can guess that the sync_bn=False is a parameter to the constructor of that class. So in the end, model is an object of that class.

Answer: If you desire to have a folder as a python module, the folder must contain an __init__.py, even if it is empty. Then you can import the rest. import os files = os.listdir(os.path.split(os.path.realpath(__file__))[0]) #get the folder's content files.remove('__init__.py') #remove __init__.py since it is empty for file in files: #loop through the files if file.endswith('.py'): #if it is a python file exec('from .{} import *'.format(file[:-3])) #import  The above code, imports every other .py files than the __init__, which is empty.  from folder1 import model as mymodel  Here folder1 is the module, and model is the object you imported from (folder) model in this case, since it is now imported to folder1's __init__.py, and now it is part of folder1 (which is a module as discussed).  model = getattr(mymodel, args.arch)(sync_bn=False)  This line is equal to: mymodel.attr, where attr is the desired attribute of the object. Can you please post more code around the getattr, since I can't tell what args.arch are refering to. As Pyzard suggested, the getattr method gets an attribute, which is a function, since it is getting called, and method is the value that this function returned. In this case sync_bn is irrevelant, but knowing more about args.arch would still help.  More about the getattr function, how import works. Better explanation of how init.py works.
Question: How can I merge/join these two dataframes ONLY on "id". Produce 3 new dataframes:  1)R1 = Merged records  2)R2 = (DF1 - Merged records) 3)R3 = (DF2 - Merged records)  Using pandas in Python. First dataframe (DF1) | id | name | |-----------|-------| | 1 | Mark | | 2 | Dart | | 3 | Julia | | 4 | Oolia | | 5 | Talia |  Second dataframe (DF2) | id | salary | |-----------|--------| | 1 | 20 | | 2 | 30 | | 3 | 40 | | 4 | 50 | | 6 | 33 | | 7 | 23 | | 8 | 24 | | 9 | 28 |  My solution for R1 =pd.merge(DF1, DF2, on='id', how='inner')  I am unsure that is the easiest way to get R2 and R3 R2 should look like | id | name | |-----------|-------| | 5 | Talia |  R3 should look like: | id | salary | |-----------|--------| | 6 | 33 | | 7 | 23 | | 8 | 24 | | 9 | 28 | 

Answer: You can turn on indicator in merge and look for the corresponding values: total_merge = df1.merge(df2, on='id', how='outer', indicator=True) R1 = total_merge[total_merge['_merge']=='both'] R2 = total_merge[total_merge['_merge']=='left_only'] R3 = total_merge[total_merge['_merge']=='right_only']   Update: Ben's suggestion would be something like this: dfs = {k:v for k,v in total_merge.groupby('_merge')}  and then you can do, for examples: dfs['both']
Question: I was taking a look at the new features that were implemented in python 3.8, and I found out about a new function numerator, denominator = x.as_integer_ratio(). They state in the documentation that it  Return a pair of integers whose ratio is exactly equal to the original integer and with a positive denominator. The integer ratio of integers (whole numbers) is always the integer as the numerator and 1 as the denominator.  Basically this code x = 10 numerator, denominator = x.as_integer_ratio() print(numerator) print(denominator)  Output 10 1  I was just wondering what is the point of having a function that will always return the same value and 1 ? I also saw that it was previously available for float which makes sense.

Answer: It appears the main reason for the changes were to implement uniformity, and typing for mypy, so that an int can be a subtype of float  msg313780 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-03-13 21:25 Goal: make int() more interoperable with float by making a float/Decimal method also available on ints. This will let mypy treat ints as a subtype of floats. See: https://mail.python.org/pipermail/python-dev/2018-March/152384.html Open question: Is this also desired for fractions.Fraction and numbers.Rational?  and digging into the pipermail it appears it helps with being able to change the domain of a type via decomposition, etc:  [Python-Dev] Symmetry arguments for API expansion Guido van Rossum guido at python.org Tue Mar 13 15:07:15 EDT 2018 Previous message (by thread): [Python-Dev] Symmetry arguments for API expansion Next message (by thread): [Python-Dev] Symmetry arguments for API expansion Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]  OK, please make it so. On Tue, Mar 13, 2018 at 11:39 AM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote:   On Mar 13, 2018, at 10:43 AM, Guido van Rossum wrote: So let's make as_integer_ratio() the standard protocol for "how to make a Fraction out of a number that doesn't implement numbers.Rational". We already have two examples of this (float and Decimal) and perhaps numpy or the sometimes proposed fixed-width decimal type can benefit from it too. If this means we should add it to int, that's fine with me.  I would like that outcome. The signature x.as_integer_ratio() -> (int, int) is pleasant to work with. The output is easy to explain, and the denominator isn't tied to powers of two or ten. Since Python ints are exact and unbounded, there isn't worry about range or rounding issues. In contrast, math.frexp(float) ->(float, int) is a bit of pain because it still leaves you in the domain of floats rather than letting you decompose to more more basic types. It's nice to have a way to move down the chain from ℚ, ℝ, or ℂ to the more basic ℤ (of course, that only works because floats and complex are implemented in a way that precludes exact irrationals). Raymond   see:  https://bugs.python.org/issue33073
Question: Dear experts i have a data set.i just want to calculate signal to noise ratio of the data. data is loaded here https://i.fluffy.cc/jwg9d7nRNDFqdzvg1Qthc0J7CNtKd5CV.html my code is given below: import numpy as np from scipy import signaltonoise import scipy.io dat=scipy.io.loadmat('./data.mat') arr=dat['dn'] snr=scipy.stats.signaltonoise(arr, axis=0, ddof=0)  but i am getting error like importError: cannot import name 'signaltonoise' from 'scipy' if it doesn't exist how to calculate snr,please suggest some other way to do it with this data set using python.

Answer: scipy.stats.signaltonoise was removed in scipy 1.0.0. You can either downgrade your scipy version or create the function yourself: def signaltonoise(a, axis=0, ddof=0): a = np.asanyarray(a) m = a.mean(axis) sd = a.std(axis=axis, ddof=ddof) return np.where(sd == 0, 0, m/sd)  Source: https://github.com/scipy/scipy/blob/v0.16.0/scipy/stats/stats.py#L1963 See the github link for the docstring. Edit: the full script would then look as follows import numpy as np import scipy.io def signaltonoise(a, axis=0, ddof=0): a = np.asanyarray(a) m = a.mean(axis) sd = a.std(axis=axis, ddof=ddof) return np.where(sd == 0, 0, m/sd) dat = scipy.io.loadmat('./data.mat') arr = dat['dn'] snr = signaltonoise(arr)
Question: When I tried to add validation_split in my LSTM model, I got this error ValueError: `validation_split` is only supported for Tensors or NumPy arrays, found: (<tensorflow.python.keras.preprocessing.sequence.TimeseriesGenerator object)  This is the code  from keras.preprocessing.sequence import TimeseriesGenerator train_generator = TimeseriesGenerator(df_scaled, df_scaled, length=n_timestamp, batch_size=1) model.fit(train_generator, epochs=50,verbose=2,callbacks=[tensorboard_callback], validation_split=0.1) ---------- ValueError: `validation_split` is only supported for Tensors or NumPy arrays, found: (<tensorflow.python.keras.preprocessing.sequence.TimeseriesGenerator object)  One reason I could think of is, to use validation_split a tensor or numpy array is expected, as mentioned in the error, however, when passing train data through TimeSeriesGenerator, it changes the dimension of the train data to a 3D array  And since TimeSeriesGenerator is mandatory to be used when using LSTM, does this means for LSTM we can't use validation_split

Answer: Your first intution is right that you can't use the validation_split when using dataset generator. You will have to understand how the functioninig of dataset generator happens. The model.fit API does not know how many records or batch your dataset has in its first epoch. As the data is generated or supplied for each batch one at a time to the model for training. So there is no way to for the API to know how many records are initially there and then making a validation set out of it. Due to this reason you cannot use the validation_split when using dataset generator. You can read it in their documentation.  Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling. This argument is not supported when x is a dataset, generator or keras.utils.Sequence instance.  You need to read the last two lines where they have said that it is not supported for dataset generator. What you can instead do is use the following code to split the dataset. You can read in detail here. I am just writing the important part from the link below. # Splitting the dataset for training and testing. def is_test(x, _): return x % 4 == 0 def is_train(x, y): return not is_test(x, y) recover = lambda x, y: y # Split the dataset for training. test_dataset = dataset.enumerate() \ .filter(is_test) \ .map(recover) # Split the dataset for testing/validation. train_dataset = dataset.enumerate() \ .filter(is_train) \ .map(recover)  I hope my answer helps you.
Question: I would like to ask you a question. Is there any algorithm/tool which can allow me to do some association between words? For example: I have the following group of sentences: (1) "My phone is on the table" "I cannot find the charger". # no reference on phone (2) "My phone is on the table" "I cannot find the phone's charger".  What I would like to do is to find a connection, probably a semantic connection, which can allow me to say that the first two sentences are talking about a topic (phone) as two terms (phone and charger) are common within it (in general). Same for the second sentence. I should have something that can connect phone to charger, in the first sentence. I was thinking of using Word2vec, but I am not sure if this is something that I can do with it. Do you have any suggestions about algorithms that I can use to determine similarity of topics (i.e. sentence which are formulated in a different way, but having same topic)?

Answer: In Python I'm pretty sure you have a Sequence Matcher that you can usee from difflib import SequenceMatcher def similar(a, b): return SequenceMatcher(None, a, b).ratio()  If you want your own algorithm I would suggest a Levenstains Distance (it calculates how many operations you need to turn one string(sentance) into another. Might be usefull.). I coded it myself in like this for two strings  edits = [[x for x in range(len(str1) + 1)] for y in range(len(str2)+ 1)] for i in range(len(str2) + 1): edits[i][0] = i for i in range(1, len(str2) + 1): for j in range(1, len(str1) + 1): if str2[i-1] == str1[j-1]: edits[i][j] = edits[i-1][j-1] else: edits[i][j] = 1 + min(edits[i-1][j-1], edits[i-1][j], edits[i][j-1]) return edits[-1][-1]  [EDIT] For you, you want to compare if the sentances are about the similar topic. I would suggest any of the following algorithms (all are pretty easy)  Jaccary Similarity K-means and Hierarchical Clustering Dendrogram Cosine Similarity
Question: I cannot figure out how to fix the following issue. Up until today I was using the following code snippet for installing RDKit in Google Colab: !wget -c https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh !chmod +x Miniconda3-latest-Linux-x86_64.sh !time bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local !time conda install -q -y -c conda-forge rdkit import sys sys.path.append('/usr/local/lib/python3.7/site-packages/')  However, today I started to get the following error: --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-2-d24c24e2d1f9> in <module>() ----> 1 from rdkit import Chem 2 import networkx as nx ModuleNotFoundError: No module named 'rdkit'  I've tried using the full Anaconda distribution instead of Miniconda, as well as changing the python version to 3.6 and 3.8 but nothing seems to work.

Answer: I think you need to specify python 3.7 when you install Miniconda (the current rdkit build supports python 3.7), the latest Miniconda version is py3.8: !wget -c https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.3-Linux-x86_64.sh !chmod +x Miniconda3-py37_4.8.3-Linux-x86_64.sh !time bash ./Miniconda3-py37_4.8.3-Linux-x86_64.sh -b -f -p /usr/local !time conda install -q -y -c conda-forge rdkit import sys sys.path.append('/usr/local/lib/python3.7/site-packages/')  https://colab.research.google.com/drive/1MAZyv3O4-TrI8c1MD4JVmwExDquaprRT?usp=sharing
Question: Im using difflib and tried to compare the two sentence and get the difference. Somewhat like this.  i have this code but instead of word by word it analyzed letter by letter. import difflib # define original text # taken from: https://en.wikipedia.org/wiki/Internet_Information_Services original = ["IIS 8.5 has several improvements related"] # define modified text edited = ["It has several improvements related"] # initiate the Differ object d = difflib.Differ() # calculate the difference between the two texts diff = d.compare(original, edited) # output the result print ('\n'.join(diff)) 

Answer: If you remove the []'s from your strings, and call .split() on them in the .compare() perhaps you'll get what you want. import difflib # define original text # taken from: https://en.wikipedia.org/wiki/Internet_Information_Services original = "IIS 8.5 has several improvements related" # define modified text edited = "It has several improvements related" # initiate the Differ object d = difflib.Differ() # calculate the difference between the two texts diff = d.compare(original.split(), edited.split()) # output the result print ('\n'.join(diff))  Output + It - IIS - 8.5 has several improvements related
Question: From the documentation:  The default transform specifies that text is in data coords, alternatively, you can specify text in axis coords (0,0 is lower-left and 1,1 is upper-right). The example below places text in the center of the axes: >>> text(0.5, 0.5, 'matplotlib', horizontalalignment='center', ... verticalalignment='center', transform=ax.transAxes)   Can I instead use both data and axis coords? For x and y respectively. Example code: import random import matplotlib.pyplot as plt values = [random.randint(2,30) for _ in range(15)] plt.violinplot(values, positions=[1]) # This might place the text outside the figure plt.gca().text(1, 30, "Text") # I would like to use axis coords for y instead of data coords. Example call # would be something like this: # text_mixed_coords(xdata=1, yaxis=0.9, "Text") plt.savefig("plot.png")  Potential results:   See also: Putting text in top left corner of matplotlib plot

Answer: This is known as a "blended transformation" you can create a blended transformation that uses the data coordinates for the x axis and axes coordinates for the y axis, like so: import matplotlib.transforms as transforms trans = transforms.blended_transform_factory(ax.transData, ax.transAxes)  In your minimal example: import random import matplotlib.pyplot as plt import matplotlib.transforms as transforms fig, ax = plt.subplots() values = [random.randint(2,30) for _ in range(15)] ax.violinplot(values, positions=[1]) # the x coords of this transformation are data, and the # y coord are axes trans = transforms.blended_transform_factory( ax.transData, ax.transAxes) ax.text(1, 0.9, "Text", transform=trans) plt.savefig("plot.png")   Also worth noting this from the matplotlib tutorial which makes it a little easier in this particular case:  Note: The blended transformations where x is in data coords and y in axes coordinates is so useful that we have helper methods to return the versions mpl uses internally for drawing ticks, ticklabels, etc. The methods are matplotlib.axes.Axes.get_xaxis_transform() and matplotlib.axes.Axes.get_yaxis_transform(). So in the example above, the call to blended_transform_factory() can be replaced by get_xaxis_transform: trans = ax.get_xaxis_transform()
Question: I have a ragged tensor, and upon trying to create a model, and use model.fit(), I get an error: TypeError: Failed to convert object of type <class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'> to Tensor. Contents: tf.RaggedTensor(values=Tensor("Cast_1:0", shape=(None,), dtype=float32), row_splits=Tensor("RaggedFromVariant_1/RaggedTensorFromVariant:0", shape=(None,), dtype=int64)). Consider casting elements to a supported type. Is this an issue with the shape of my data? Perhaps the data altogether? Maybe I need to use a sparse or dense tensor instead? Here is my full traceback error:  As well as my data: 

Answer: Every RaggedTensor has an associated to_tensor() method, call that. Check your input data transformations, that is where the RaggedTensor is getting created. Some ops return RaggedTensor.