eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I discovered today that I can access elements in a queryset by referencing them with an index, i.e. queryset[n]. However, immediately after I discovered that queryset[0] does not return the same record as queryset.first(). Why is this, and is one of those more "correct"? (I know that .first() is faster, but other than that) Python 3.7.4 django 1.11.20

Answer: There is a small semantical difference between qs[0] and qs.first(). If you did not specify an order in the queryset yourself, then Django will order the queryset itself by primary key before fetching the first element. Furthermore .first() will return None if the queryset is empty. Whereas qs[0] will raise an IndexError. The claim that .first() is faster however is not True. In fact if you use qs[n], then Django will, behind the curtains fetch the record by slicing with qs[n:n+1], hence it will, given the database backend supports this, make a query with LIMIT 1 OFFSET n, and thus fetch one record, just like .first() will do. If the queryset is already retrieved, it will furthermore make no extra queries at all, since the data is already cached. You can see the implementation on GitHub:  def first(self): """ Returns the first object of a query, returns None if no match is found. """ objects = list((self if self.ordered else self.order_by('pk'))[:1]) if objects: return objects[0] return None As you can see, if the queryset is already ordered (self.ordered is True, then we take self[:1], check if there is a record, and if so return it. If not we return None. The code for retrieving an item at a specific index is more cryptic. It essentially will set the limits from k to k+1, materialize the item, and return the first item, as we can see in the source code [GitHub]:  def __getitem__(self, k): """ Retrieves an item or slice from the set of results. """ if not isinstance(k, (slice,) + six.integer_types): raise TypeError assert ((not isinstance(k, slice) and (k >= 0)) or (isinstance(k, slice) and (k.start is None or k.start >= 0) and (k.stop is None or k.stop >= 0))), \ "Negative indexing is not supported." if self._result_cache is not None: return self._result_cache[k] if isinstance(k, slice): qs = self._clone() if k.start is not None: start = int(k.start) else: start = None if k.stop is not None: stop = int(k.stop) else: stop = None qs.query.set_limits(start, stop) return list(qs)[::k.step] if k.step else qs qs = self._clone() qs.query.set_limits(k, k + 1) return list(qs)[0]
Question: When trying lemmatize in Spanish a csv with more than 60,000 words, SpaCy does not correctly write certain words, I understand that the model is not 100% accurate. However, I have not found any other solution, since NLTK does not bring a Spanish core. A friend tried to ask this question in Spanish Stackoverflow, however, the community is quite small compared with this community, and we got no answers about it. code: nlp = spacy.load('es_core_news_sm') def lemmatizer(text): doc = nlp(text) return ' '.join([word.lemma_ for word in doc]) df['column'] = df['column'].apply(lambda x: lemmatizer(x))  I tried to lemmatize certain words that I found wrong to prove that SpaCy is not doing it correctly: text = 'personas, ideas, cosas' # translation: persons, ideas, things print(lemmatizer(text))  # Current output: personar , ideo , coser # translation: personify, ideo, sew # The expected output should be: persona, idea, cosa # translation: person, idea, thing 

Answer: Unlike the English lemmatizer, spaCy's Spanish lemmatizer does not use PoS information at all. It relies on a lookup list of inflected verbs and lemmas (e.g., ideo idear, ideas idear, idea idear, ideamos idear, etc.). It will just output the first match in the list, regardless of its PoS. I actually developed spaCy's new rule-based lemmatizer for Spanish, which takes PoS and morphological information (such as tense, gender, number) into account. These fine-grained rules make it a lot more accurate than the current lookup lemmatizer. It will be released soon! Meanwhile, you can maybe use Stanford CoreNLP or FreeLing.
Question: I have a function that takes an array as input and does some computation on it. The input array may or may not be a numpy ndarray (may be a list, pandas object, etc).  In the function, I convert the input array (regardless of its type) to a numpy ndarray. But this step may be computationally expensive for large arrays, especially if the function is called multiple times in a for loop.  Hence, I want to convert the input array to numpy ndarray ONLY if it is not already a numpy ndarray.  How can I do this? import numpy as np def myfunc(array): # Check if array is not already numpy ndarray # Not correct way, this is where I need help if type(array) != 'numpy.ndarray': array = np.array(array) # The computation on array # Do something with array new_array = other_func(array) return new_array 

Answer: It is simpler to use asarray: def myfunc(arr): arr = np.asarray(arr) # The computation on array # Do something with array new_array = other_func(arr) return new_array  If arr is already an array, asarray does not make a copy, so there's no penalty to passing it through asarray. Let numpy do the testing for you. numpy functions often pass their inputs through asarray (or variant) just make sure the type is what they expect.
Question: In python I have a Random dictionary where I use tuple as a key and each is mapped to some value. Sample Random_Dict = { (4, 2): 1, (2, 1): 3, (2, 0): 7, (1, 0): 8 }  example in above key: (4,2) value: 1 I am attempting to replicate this in Javascript world This is what I came up with const randomKeys = [[4, 2], [2, 1], [2, 0], [1, 0] ] const randomMap = {} randomMap[randomKeys[0]] = 1 randomMap[randomKeys[1]] = 3 randomMap[randomKeys[2]] = 7 randomMap[randomKeys[3]] = 8 randomMap[[1, 2]] = 3  I am wondering if this is the most effective way. I almost wonder if i should do something like holding two numbers in one variable so that way i can have a dictionary in JS that maps 1:1. Looking for suggestions and solutions that are better

Answer: You can use a Map to map sets of 2 arbitrary values. In the following snippet the keys can be 'tuples' (1), or any other data type, and the values can be as well:   const values = [ [ [4, 2], 1], [ [2, 1], 3], [ [2, 0], 7], [ [1, 0], 9], ]; const map = new Map(values); // Get the number corresponding a specific 'tuple' console.log( map.get(values[0][0]) // should log 1 ); // Another try: console.log( map.get(values[2][0]) // should log 7 );     Note that the key equality check is done by reference, not by value equivalence. So the following logs undefined for the above example, although the given 'key' is also an array of the shape [4, 2] just like one of the Map keys: console.log(map.get([4, 2]));   (1) Tuples don't technically exist in Javascript. The closest thing is an array with 2 values, as I used in my example.
Question: I have a problem when I read a .csv and set the 'Column A' as index column. df = pd.read_csv(index_col = 'Column A') print(df.colums)  However, I cannot access 'Column A' anymore. I still want to use it as one column to access its date.  Can anyone help? 

Answer: I found this is very straightforward: just setting index as a column. df['index1'] = df.index
Question:  I'm not quite sure if my question is a duplicate, but I wasn't able to find something helping me in my case. Set Up I've built a frontend webpage which contains a couple of services, for example show some timeseries and other information about my system. The website is build with the react framework and so using javascript in general.  Now I want to do some calculations about the timeseries for example calculate the similarity and other features of my sensordata. For that I'm using python which offers me a lot of libraries I've used for a long time and are easy to use.What I'm looking for: I'm looking for a very simple way to call my backend-timeseries-analysis-python script from the react GUI passing some variables like the length of the series. Also I want to process the returned values and safe the current values needed for normalization (like max,min) for further calculations.  So the procedure would look like the following: 1) Type value in react frontend input box 2) react/javascript calls pythonscript/ initialize a class and passes variables to class 3) python calculates similarity of sensor data 4) python returns similarity values to frontend and saves classes for later call 5) react displays returned values 6) react/javascript calls pythonscript 7) python compares latest data to past data and refreshs treshholds(like max, min) 8) python calculates similarity of sensor data 9) continue..  Thanks for your help!  

Answer: You can expose your Python scripts on a REST API which will be called by your React frontend. Database connection will be made by this API, and the response is sent to your frontend. See Flask (very simple for small projects) or even Django to build Python APIs.
Question: I'm having trouble understanding why importing a global variable from another module works as expected when using import, but when using from x import * the global variable doesn't appear to update within its own module Imagine I have 2 files, one.py: def change(value): global x x = value x = "start"  and two.py: from one import * print x change("updated") print x  I'd expect: start updated  But I get... start start  If I import the module normally it works as expected import one print one.x one.change("updated") print one.x  Result... start updated  Given that I can't change ony.py's use of global variables (not my fault), and two.py is meant to be a sort of wrapper* around one.py, I'd really like to avoid using the one. namespace throughout two.py for the sake of one stubborn variable. If it's not possible a novice-level explantion of what's going on might help me avoid getting stuck like this again. I undertand that one.x is getting updated, but two.x isn't respecting the updates, but I don't know why.

Answer: You can think of a package as a dict. Every function and variable in a package is listed as a key in that dict which you can view using globals().  When you import an object from another package, you copy a reference to the object into your own package under a name (usually the same, different if you import <var> as <name>). By setting the value in the other package, you overwrite the key in that package with a new value, but you leave your reference pointing to the old one. An analogy with dicts We can demonstrate the process using dicts as an analogy: # Our 'packages' one = {'x': 'start'} two = {} # Equivalent of our import two['x'] = one['x'] # Variable updated in `one' one['x'] = 'updated' # ... and accessed in `two` print(two['x']) # Still 'start'  We would not expect the two dict to be updated just because we overwrote a value in one. What you can do about it You can modify the object as long as you don't break the pointer by overwriting the variable. For example if x was a dict, you could change a value inside the dict and the two variables would still point to the same object. Alternatively you could attach a variables to the function like this: def change(value): change.x = value  This does the same work by ensuring we are mutating the same object. A better answer yet might be to wrap both items in an object if they need to travel together: class Changer: x = 'start' @classmethod def change(cls, value): cls.x = value  At this point however, we could modify the value directly as an attribute: Changer.x = 'updated'  Which might be the simplest.
Question: I have always been under impression that Pandas uses hashing when indexing the rows in a dataframe such that the operations like df.loc[some_label] is O(1). However, I just realized today that this is not the case, at least for multi-indexed dataframe. As pointed out in the document, "Indexing will work even if the data are not sorted, but will be rather inefficient (and show a PerformanceWarning)". Some articles I found seem to suggest, for multiindex dataframe, Pandas is using binary-search based indexing if you have called sort_index() on the dataframe; otherwise, it just linearly scans the rows. My question are  Does single-indexed dataframe use hash-based indexing or not? If not for question 1, does it use binary-search when sort_index() has been called, and linear scan otherwise, like in the case of multi-indexed dataframe? If yes for question 1, why Pandas choose to not use hash-based indexing for multi-index as well? 

Answer: Does a single-indexed DataFrame use hash-based indexing?  No, Pandas does not use hash-based indexing for single-indexed DataFrames. Instead, it relies on array-based lookups or binary search when the index is sorted. If the index is unsorted, Pandas performs a linear scan, which is less efficient.  If the DataFrame is sorted using sort_index(), Pandas can leverage a binary search to achieve faster lookups. Without sorting, lookups default to a linear scan.  Hash-based indexing is more challenging for multi-indexes due to the hierarchical nature of the index. Instead, Pandas relies on binary search (for sorted indexes) or linear scan (for unsorted indexes) because these methods handle hierarchical indexing efficiently. Hash-based indexing would introduce additional overhead and complexity when working with multiple levels.
Question: I'm trying to build a CNN but I get this error: ---> 52 x = x.view(x.size(0), 5 * 5 * 16) RuntimeError: shape '[16, 400]' is invalid for input of size 9600  It's not clear for me what the inputs of the 'x.view' line should be. Also, I don't really understand how many times I should have this 'x.view' function in my code. Is it only once, after the 3 convolutional layers and 2 linear layers? Or is it 5 times, one after every layer? Here's my CNN code: import torch.nn.functional as F # Convolutional neural network class ConvNet(nn.Module): def __init__(self, num_classes=10): super(ConvNet, self).__init__() self.conv1 = nn.Conv2d( in_channels=3, out_channels=16, kernel_size=3) self.conv2 = nn.Conv2d( in_channels=16, out_channels=24, kernel_size=4) self.conv3 = nn.Conv2d( in_channels=24, out_channels=32, kernel_size=4) self.dropout = nn.Dropout2d(p=0.3) self.pool = nn.MaxPool2d(2) self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(512, 10) self.final = nn.Softmax(dim=1) def forward(self, x): print('shape 0 ' + str(x.shape)) x = F.max_pool2d(F.relu(self.conv1(x)), 2) x = self.dropout(x) print('shape 1 ' + str(x.shape)) x = F.max_pool2d(F.relu(self.conv2(x)), 2) x = self.dropout(x) print('shape 2 ' + str(x.shape)) # x = F.max_pool2d(F.relu(self.conv3(x)), 2) # x = self.dropout(x) x = F.interpolate(x, size=(5, 5)) x = x.view(x.size(0), 5 * 5 * 16) x = self.fc1(x) return x net = ConvNet()  Can someone help me understand the problem? The output of x.shape is: shape 0 torch.Size([16, 3, 256, 256]) shape 1 torch.Size([16, 16, 127, 127]) shape 2 torch.Size([16, 24, 62, 62]) Thanks.

Answer: This means that instead the product of the channel and spatial dimensions is not 5*5*16. To flatten the tensor, replace x = x.view(x.size(0), 5 * 5 * 16) with: x = x.view(x.size(0), -1)
Question: I was struck by this default behavior of f-strings in python 3.7.2: >> number = 0.0000001 >> string = f"Number: {number}" >> print(string) Number: 1e-07  What I expected was: Number: 0.0000001 This is very annoying especially for creation of filenames. How can I disable this automatic conversion into the scientific notation? And why is it enabled in the first place? Basically the opposite of this question. Edit: I would like to avoid setting a fixed length for the float via {number:.8f} since my numbers have different lengths and I don't want to have any trailing zeros. I want to use the f-strings to make filenames automatically, like this:  filename = f"number_{number:.10f}_other_number_{other_number:.10f}.json"  I am looking for a simple modifier that can disable the automatic scientific notation while keeping the original precision of the float.

Answer: After checking other sources, I found numpy.format_float_positional which works very nicely here: import numpy as np number = 0.0000001 string = f"Number: {np.format_float_positional(number)}"
Question: Is there a way to kill uvicorn cleanly? I.e., I can type ^C at it, if it is running in the foreground on a terminal. This causes the uvivorn process to die and all of the worker processes to be cleaned up. (I.e., they go away.) On the other hand, if uvicorn is running in the background without a terminal, then I can't figure out a way to kill it cleanly. It seems to ignore SIGTERM, SIGINT, and SIGHUP. I can kill it with SIGKILL (i.e. -9), but then the worker processes remain alive, and I have to track all the worker processes down and kill them too. This is not ideal. I am using uvicorn with CPython 3.7.4, uvivorn version 0.11.2, and FastAPI 0.46.0 on Red Hat Enterprise Linux Server 7.3 (Maipo).

Answer: That's because you're running uvicorn as your only server. uvicorn is not a process manager and, as so, it does not manage its workers life cycle. That's why they recommend running uvicorn using gunicorn+UvicornWorker for production. That said, you can kill the spawned workers and trigger it's shutdown using the script below: $ kill $(pgrep -P $uvicorn_pid)  The reason why this works but not the kill on the parent pid is because when you ^C something, the signal is transmitted throughout all of its spawned processes that are attached to the stdin.
Question: I have a project in which I have to install from git+https: I can make it to work in this way: virtualenv -p python3.5 bla . bla/bin/activate pip install numpy # must have numpy before the following pkg... pip install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'  However, I want to use it in a setup.py file in install_requires: from setuptools import setup setup(install_requires='git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI', setup_requires='numpy')  and then, pip install -e . from the dir containing the setup.py This doesn't work due to parse error:   Complete output (1 lines): error in bla_bla setup command: 'install_requires' must be a string or list of strings containing valid project/version requireme nt specifiers; Invalid requirement, parse error at "'+https:/'" ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.  The error doesn't occur if I install using pip install -r requires.txt (assuming I have the same string in that file) and not when using direct pip install git+...... How to fix this parsing error? What I've tried so far:  wrapping the string with " / """ / ' / ''' adding 'r' before the string 

Answer: install_requires must be a string or a list of strings with names and optionally URLs to get the package from: install_requires=[ 'pycocotools @ git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI' ]  See https://pip.pypa.io/en/stable/reference/requirement-specifiers/ and https://www.python.org/dev/peps/pep-0440/#direct-references This requires pip install including pip install . and doesn't work with python setup.py install.
Question: For traditional Python projects with a setup.py, there are various ways of ensuring that the version string does not have to be repeated throughout the code base. See PyPA's guide on "Single-sourcing the package version" for a list of recommendations. Many are trying to move away from setup.py to setup.cfg (probably under the influence of PEP517 and PEP518; setup.py was mostly used declaratively anyway, and when there was logic in setup.py, it was probably for the worse.) This means that most the suggestions won't work anymore since setup.cfg cannot contain "code". How can I single-source the package version for Python projects that use setup.cfg?

Answer: There are a couple of ways to do this (see below for the project structure used in these examples): 1. setup.cfg [metadata] version = 1.2.3.dev4  src/my_top_level_package/__init__.py import importlib.metadata __version__ = importlib.metadata.version('MyProject')  2. setup.cfg [metadata] version = file: VERSION.txt  VERSION.txt 1.2.3.dev4  src/my_top_level_package/__init__.py import importlib.metadata __version__ = importlib.metadata.version('MyProject')  3. setup.cfg [metadata] version = attr: my_top_level_package.__version__  src/my_top_level_package/__init__.py __version__ = '1.2.3.dev4'  And more... There are probably other ways to do this, by playing with different combinatons.  References:  https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html https://docs.python.org/3/library/importlib.metadata.html   Structure assumed in the previous examples is as follows... MyProject ├── setup.cfg ├── setup.py └── src └── my_top_level_package └── __init__.py  setup.py #!/usr/bin/env python3 import setuptools if __name__ == '__main__': setuptools.setup( # see 'setup.cfg' )  setup.cfg [metadata] name = MyProject # See above for the value of 'version = ...' [options] package_dir = = src packages = find: [options.packages.find] where = src  $ cd path/to/MyProject $ python3 setup.py --version 1.2.3.dev4 $ python3 -m pip install . # ... $ python3 -c 'import my_top_level_package; print(my_top_level_package.__version__)' 1.2.3.dev4 $ python3 -V Python 3.6.9 $ python3 -m pip list Package Version ------------- ---------- MyProject 1.2.3.dev4 pip 20.0.2 pkg-resources 0.0.0 setuptools 45.2.0 wheel 0.34.2 zipp 3.0.0
Question: I am training a binary classifier on a dataset of cats and dogs: Total Dataset: 10000 images Training Dataset: 8000 images Validation/Test Dataset: 2000 images The Jupyter notebook code: # Part 2 - Fitting the CNN to the images train_datagen = ImageDataGenerator(rescale = 1./255, shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True) test_datagen = ImageDataGenerator(rescale = 1./255) training_set = train_datagen.flow_from_directory('dataset/training_set', target_size = (64, 64), batch_size = 32, class_mode = 'binary') test_set = test_datagen.flow_from_directory('dataset/test_set', target_size = (64, 64), batch_size = 32, class_mode = 'binary') history = model.fit_generator(training_set, steps_per_epoch=8000, epochs=25, validation_data=test_set, validation_steps=2000)  I trained it on a CPU without a problem but when I run on GPU it throws me this error: Found 8000 images belonging to 2 classes. Found 2000 images belonging to 2 classes. WARNING:tensorflow:From <ipython-input-8-140743827a71>:23: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version. Instructions for updating: Please use Model.fit, which supports generators. WARNING:tensorflow:sample_weight modes were coerced from ... to ['...'] WARNING:tensorflow:sample_weight modes were coerced from ... to ['...'] Train for 8000 steps, validate for 2000 steps Epoch 1/25 250/8000 [..............................] - ETA: 21:50 - loss: 7.6246 - accuracy: 0.5000 WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 200000 batches). You may need to use the repeat() function when building your dataset. 250/8000 [..............................] - ETA: 21:52 - loss: 7.6246 - accuracy: 0.5000  I would like to know how to use the repeat() function in keras using Tensorflow 2.0?

Answer: Your problem stems from the fact that the parameters steps_per_epoch and validation_steps need to be equal to the total number of data points divided by the batch_size. Your code would work in Keras 1.X, prior to August 2017. Change your model.fit() function to: history = model.fit_generator(training_set, steps_per_epoch=int(8000/batch_size), epochs=25, validation_data=test_set, validation_steps=int(2000/batch_size))  As of TensorFlow 2.1, fit_generator() is being deprecated. You can use .fit() method also on generators. TensorFlow >= 2.1 code: history = model.fit(training_set.repeat(), steps_per_epoch=int(8000/batch_size), epochs=25, validation_data=test_set.repeat(), validation_steps=int(2000/batch_size))  Notice that int(8000/batch_size) is equivalent to 8000 // batch_size (integer division)