eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an issue that I can't wrap my head around. I have an API service built using FastAPI, and when I try to call any endpoint from another Python script on my local machine, the response takes 2+ seconds. When I send the same request through cURL or the built-in Swagger docs, the response is nearly instant. The entire server script is this: from fastapi import FastAPI import uvicorn app = FastAPI() @app.get("/") async def root(): return {"message": "Hello World"} if __name__ == '__main__': uvicorn.run(app, host='0.0.0.0', port=8000)  I then call it from a test script using HTTPX. I also tried with the requests package, and it is the same result. import httpx r = httpx.get('http://localhost:8000/') print(r.elapsed)  This prints something like: 0:00:02.069705 I then do the same thing using cURL: curl -w "@curl-format.txt" -o /dev/null -X 'GET' 'http://localhost:8000/'  This prints:  % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 32 100 32 0 0 941 0 --:--:-- --:--:-- --:--:-- 969 time_namelookup: 0.006436s time_connect: 0.006747s time_appconnect: 0.000000s time_pretransfer: 0.006788s time_redirect: 0.000000s time_starttransfer: 0.034037s ---------- time_total: 0.034093s  The issue isn't what the endpoint does, but rather that it doesn't even start executing for 2 seconds. I have a debugger running on the endpoint, and the first line only gets executed after those 2 seconds. I tried to inspect the request to see whether there are any headers or similar in the request that could slow it down, but nothing. When I try again with the headers generated by HTTPX, it still executes fast: curl -w "@curl-format.txt" -o /dev/null -X 'GET' \ 'http://localhost:8000/events' \ -H 'accept: */*' \ -H 'host: localhost:8000' \ -H 'accept-encoding: gzip, deflate' \ -H 'connection: keep-alive' \ -H 'user-agent: python-httpx/0.20.0'  Here is a screenshot of the request in PyCharm, it unfortunately can't be dumped to JSON directly.  I'm starting to think that it has something to do with Uvicorn and how it runs the app, but I can't figure out why.

Answer: Try using „127.0.0.1“ instead of „localhost“ to refer to your machine. I once had a similar issue, where the DNS lookup for localhost on windows was taking half a second or longer. I don‘t have an explanation for that behaviour, but at least one other person on SO seems to have struggled with it as well…
Question: I’ve seen code that declares abstract methods that actually have a non-trivial body. What is the point of this since you have to implement in any concrete class anyway? Is it just to allow you to do something like this?  def method_a(self): super(self).method_a() 

Answer: I've used this before in cases where it was possible to have the concrete implementation, but I wanted to force subclass implementers to consider if that implementation is appropriate for them. One specific example: I was implementing an abstract base class with an abstract factory method so subclasses can define their own __init__ function but have a common interface to create them. It was something like class Foo(ABC): def __init__(self, a, b, c): self.a = a self.b = b self.c = c @classmethod @abstractmethod def from_args(cls, a, b, c) -> "Foo": return cls(a, b, c)  Subclasses usually only need a subset of the arguments. When testing, it's cumbersome to access the framework which actually uses this factory method, so it's likely someone will forget to implement the from_args factory function since it wouldn't come up in their usual testing. Making it an abstractmethod would make it impossible to initialize the class without first implementing it, and will definitely come up during normal testing.
Question: I've got a time series of sunspot numbers, where the mean number of sunspots is counted per month, and I'm trying to use a Fourier Transform to convert from the time domain to the frequency domain. The data used is from https://wwwbis.sidc.be/silso/infosnmtot. The first thing I'm confused about is how to express the sampling frequency as once per month. Do I need to convert it to seconds, eg. 1/(seconds in 30 days)? Here's what I've got so far: fs = 1/2592000 #the sampling frequency is 1/(seconds in a month) fourier = np.fft.fft(sn_value) #sn_value is the mean number of sunspots measured each month freqs = np.fft.fftfreq(sn_value.size,d=fs) power_spectrum = np.abs(fourier) plt.plot(freqs,power_spectrum) plt.xlim(0,max(freqs)) plt.title("Power Spectral Density of the Sunspot Number Time Series") plt.grid(True)   I don't think this is correct - namely because I don't know what the scale of the x-axis is. However I do know that there should be a peak at (11years)^-1. The second thing I'm wondering from this graph is why there seems to be two lines - one being a horizontal line just above y=0. It's more clear when I change the x-axis bounds to: plt.xlim(0,1).  Am I using the fourier transform functions incorrectly?

Answer: You can use any units you want. Feel free to express your sampling frequency as fs=12 (samples/year), the x-axis will then be 1/year units. Or use fs=1 (sample/month), the units will then be 1/month. The extra line you spotted comes from the way you plot your data. Look at the output of the np.fft.fftfreq call. The first half of that array contains positive values from 0 to 1.2e6 or so, the other half contain negative values from -1.2e6 to almost 0. By plotting all your data, you get a data line from 0 to the right, then a straight line from the rightmost point to the leftmost point, then the rest of the data line back to zero. Your xlim call makes it so you don’t see half the data plotted. Typically you’d plot only the first half of your data, just crop the freqs and power_spectrum arrays.
Question: Please refer to below execution - import sys _list = [2,55,87] print(f'1 - Memory used by Python List - {sys.getsizeof(_list)}') narray = np.array([2,55,87]) size = narray.size * narray.itemsize print(f'2 - Memory usage of np array using itemsize - {size}') print(f'3 - Memory usage of np array using getsizeof - {sys.getsizeof(narray)}')  Here is what I get in result 1 - Memory used by Python List - 80 2 - Memory usage of np array using itemsize - 12 3 - Memory usage of np array using getsizeof - 116  One way of calculation suggests numpy array is consuming way too less memory but other says it is consuming more than regular python list? Shouldn't I be using getSizeOf with numpy array. What I am doing wrong here? Edit - I just checked, an empty python list is consuming 56 bytes whereas an empty np array 104. Is this space being used in pointing to associated built-in methods and attributes?

Answer: The calculation using: size = narray.size * narray.itemsize  does not include the memory consumed by non-element attributes of the array object. This can be verified by the documentation of ndarray.nbytes: >>> x = np.zeros((3,5,2), dtype=np.complex128) >>> x.nbytes 480 >>> np.prod(x.shape) * x.itemsize 480  In the above link, it can be read that ndarray.nbytes:  Does not include memory consumed by non-element attributes of the array object.  Note that from the code above you can conclude that your calculation excludes non-element attributes given that the value is equal to the one from ndarray.nbytes. A list of the non-element attributes can be found in the section Array Attributes, including here for completeness:  ndarray.flags Information about the memory layout of the array. ndarray.shape Tuple of array dimensions. ndarray.strides Tuple of bytes to step in each dimension when traversing an array. ndarray.ndim Number of array dimensions. ndarray.data Python buffer object pointing to the start of the array’s data. ndarray.size Number of elements in the array. ndarray.itemsize Length of one array element in bytes. ndarray.nbytes Total bytes consumed by the elements of the array. ndarray.base Base object if memory is from some other object.  With regards to sys.getsizeof it can be read in the documentation (emphasis mine) that:  Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.
Question: I am trying to plot a scatter graph on some data with grouping. They are grouped by the column group and I want them to have different marker styles based on the group. Minimal working code import matplotlib.pyplot as plt import pandas as pd import numpy as np colors = ['r','g','b','y'] markers = ['o', '^', 's', 'P'] df = pd.DataFrame() df["index"] = list(range(100)) df["data"] = np.random.randint(100, size=100) df["group"] = np.random.randint(4, size=100) df["color"] = df.apply(lambda x: colors[x["group"]], axis=1) df["marker"] = df.apply(lambda x: markers[x["group"]], axis=1) plt.scatter(x=df["index"], y=df["data"], c=df["color"]) # What I thought would have worked # plt.scatter(x=df["index"], y=df["data"], c=df["color"], marker=df["marker"]) plt.show()   What I want I want the groups to have different marker styles as well. For example the red entries will have marker "o" (big dot), green entries with marker "^" (upward triangle) and so on. What I tried I thought plt.scatter(x=df["index"], y=df["data"], c=df["color"], marker=df["marker"])  would have worked but nope... TypeError: 'Series' objects are mutable, thus they cannot be hashed  I can for loop over the DataFrame and group the entries by their group. Then plot them with the marker argument set with the list defined (like plt.scatter(..., marker=markers[group]). That would result in 4 plt.scatter(...) as there are 4 groups in total. But that is ugly IMO to loop through a DataFrame row by row and I strongly believe there is a better way. Thanks in advance!

Answer: matplotlib  that is ugly IMO to loop through a DataFrame row by row and I strongly believe there is a better way  With matplotlib, I don't think there is a better way than to loop. Note that if you groupby the markers, it does not loop row by row, just group by group (so 4 times in this case). This will call plt.scatter 4 times (once per marker): for marker, d in df.groupby('marker'): plt.scatter(x=d['index'], y=d['data'], c=d['color'], marker=marker, label=marker) plt.legend()    seaborn As r-beginners commented, sns.scatterplot supports multiple markers via style: sns.scatterplot(x=df['index'], y=df['data'], c=df['color'], style=df['marker'])
Question: I want to execute a python script to set some environment variables in GitHub actions. I want to use those environment variables later in my GitHub actions steps. My python script looks like: new_ver = get_version_from_commit(commit_msg) if new_ver: if new_ver == "false": os.environ["SHOULD_PUSH"] = "0" print("Not pushing the image to k8s") exit(0) else: new_tag = app_name + ":" + str(new_ver) os.environ["DOCKER_IMAGE_TAG"] = new_tag os.environ["SHOULD_PUSH"] = "1" print("New tag: " + new_tag) exit(0)  Part of my GitHub actions file, after the execution of the above python script looks like: - name: Print env var run: echo ${{ env.DOCKER_IMAGE_TAG }} - name: Build and push id: docker_build uses: docker/build-push-action@v2 with: push: true tags: ${{ secrets.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_TAG }}  But using os.environ won't expose the environment variable outside of the python process. How can I fix this ?

Answer: You cannot set environment variables directly. Instead, you need to write your environment variables into a file, whose name you can get via $GITHUB_ENV. In a simple workflow step, you can append it to the file like so (from the docs): echo "{name}={value}" >> $GITHUB_ENV  In python, you can do it like so: import os env_file = os.getenv('GITHUB_ENV') with open(env_file, "a") as myfile: myfile.write("MY_VAR=MY_VALUE")  Given this python script, you can set and use your new environment variable like the following: - run: python write-env.py - run: echo ${{ env.MY_VAR }}
Question: I have a model which looks as follows: IMG_WIDTH = IMG_HEIGHT = 224 class AlexNet(nn.Module): def __init__(self, output_dim): super(AlexNet, self).__init__() self._to_linear = None self.x = torch.randn(3, IMG_WIDTH, IMG_HEIGHT).view(-1, 3, IMG_WIDTH, IMG_HEIGHT) self.features = nn.Sequential( nn.Conv2d(3, 64, 3, 2, 1), # in_channels, out_channels, kernel_size, stride, padding nn.MaxPool2d(2), nn.ReLU(inplace=True), nn.Conv2d(64, 192, 3, padding=1), nn.MaxPool2d(2), nn.ReLU(inplace=True), nn.Conv2d(192, 384, 3, padding=1), nn.MaxPool2d(2), nn.ReLU(inplace=True), nn.Conv2d(384, 256, 3, padding=1), nn.MaxPool2d(2), nn.ReLU(inplace=True), nn.Conv2d(256, 512, 3, padding=1), nn.ReLU(inplace=True), nn.Conv2d(512, 256, 3, padding=1), nn.MaxPool2d(2), nn.ReLU(inplace=True) ) self.conv(self.x) self.classifier = nn.Sequential( nn.Dropout(.5), nn.Linear(self._to_linear, 4096), nn.ReLU(inplace=True), nn.Dropout(.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True), nn.Linear(4096, output_dim), ) def conv(self, x): x = self.features(x) if self._to_linear is None: self._to_linear = x.shape[1] * x.shape[2] * x.shape[3] return x def forward(self, x): x = self.conv(x) h = x.view(x.shape[0], -1) x = self.classifier(h) return x, h  Here is my optimizer and loss functions: optimizer = torch.optim.Adam(model.parameters()) criterion = nn.BCEWithLogitsLoss().to(device)  Here is my train and evaluate functions: def train(model, iterator, optimizer, criterion, device): epoch_loss, epoch_acc = 0, 0 model.train() for (x, y) in iterator: # features and labels to the device x = x.to(device) y = y.to(device).long() # Zero the gradients optimizer.zero_grad() y_pred, _ = model(x) # Calculate the loss and accuracy loss = criterion(y_pred.squeeze(), y) acc = binary_accuracy(y_pred, y) # Backward propagate loss.backward() # Update the weights optimizer.step() epoch_loss +=loss.item() epoch_acc += acc.item() return epoch_loss/len(iterator), epoch_acc/len(iterator) def evaluate(model, iterator, criterion, device): epoch_loss, epoch_acc = 0, 0 model.eval() with torch.no_grad(): for (x, y) in iterator: x = x.to(device) y = y.to(device).long() y_pred, _ = model(x) loss = criterion(y_pred, y) acc = binary_accuracy(y_pred, y) epoch_loss += loss.item() epoch_acc += acc.item() return epoch_loss/len(iterator), epoch_acc/len(iterator)  This is the error that I'm getting: RuntimeError: result type Float can't be cast to the desired output type Long  What may be possibly my problem because I have tried to convert my labels to long tensors as follows: y = y.to(device).long()  But it seems not to work.

Answer: I was getting the same error doing this: loss_fn(output, target)  where the output was Tensor torch.float32 and target was Tensor torch.int64. What solved this problem was calling the loss function like this: loss_fn(output, target.float())
Question: say I have two dataframes with multiindices, where one of the indices is deeper than the other. Now I want to select only those rows from the one (deeper) dataframe where their partial index is included in the other dataframe. Example input: df = pandas.DataFrame( { "A": ["a1", "a1", "a1", "a2", "a2", "a2"], "B": ["b1", "b1", "b2", "b1", "b2", "b2"], "C": ["c1", "c2", "c1", "c1", "c1", "c2"], "V": [1, 2, 3, 4, 5, 6], } ).set_index(["A", "B", "C"]) df2 = pandas.DataFrame( { "A": ["a1", "a1", "a2", "a2"], "B": ["b1", "b3", "b1", "b3"], "X": [1, 2, 3, 4] } ).set_index(["A", "B"])  Visual:  V A B C a1 b1 c1 1 c2 2 b2 c1 3 a2 b1 c1 4 b2 c1 5 c2 6 X A B a1 b1 1 b3 2 a2 b1 3 b3 4  Desired output: result = pandas.DataFrame( { "A": ["a1", "a1", "a2"], "B": ["b1", "b1", "b1"], "C": ["c1", "c2", "c1"], "V": [1, 2, 4], } ).set_index(["A", "B", "C"])  Visual:  V A B C a1 b1 c1 1 c2 2 a2 b1 c1 4  I tried df.loc[df2.index] and df.loc[df.index.intersection(df2.index)] but that does not work. I guess I could do df.join(df2, how="inner") and afterwards remove all the columns of df2 that were added, but that is cumbersome. Or is there a way to take away all the columns of df2? I would appreciate any help.

Answer: One option is to use isin on the specific labels common to both, and use the resulting boolean to filter df: df.loc[df.index.droplevel('C').isin(df2.index)] V A B C a1 b1 c1 1 c2 2 a2 b1 c1 4
Question: I made a drowsiness detector for driving using django. I tried deploying it on heroku and everything seems to be working fine except as soon as I try to open the camera, I get the error: Exception Value: Failed loading libasound.so.2: libasound.so.2: cannot open shared object file: No such file or directory  All the other pages are working fine, you can try it out here (login credentials username: temp password: Temp@123). As soon as you click on Start Driving it throws this exception. The app is working fine in localhost. Please help! Log after git push heroku master: Enumerating objects: 5, done. Counting objects: 100% (5/5), done. Delta compression using up to 12 threads Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 288 bytes | 288.00 KiB/s, done. Total 3 (delta 1), reused 0 (delta 0) remote: Compressing source files... done. remote: Building source: remote: remote: -----> Building on the Heroku-20 stack remote: -----> Using buildpack: heroku/python remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt remote: -----> No change in requirements detected, installing from cache remote: -----> Using cached install of python-3.8.12 remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0 remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip remote: -----> $ python manage.py collectstatic --noinput remote: 130 static files copied to '/tmp/build_fa3901e7/staticfiles'. remote: remote: -----> Discovering process types remote: Procfile declares types -> web remote: remote: -----> Compressing... remote: Done: 319.5M remote: -----> Launching... remote: ! Warning: Your slug size (319 MB) exceeds our soft limit (300 MB) which may affect boot time. remote: Released v13 remote: https://dontsleepapp.herokuapp.com/ deployed to Heroku remote: remote: Verifying deploy... done. To https://git.heroku.com/dontsleepapp.git  After heroku logs --tail 2021-11-26T06:07:53.697167+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response 2021-11-26T06:07:53.697168+00:00 app[web.1]: response = wrapped_callback(request, *callback_args, **callback_kwargs) 2021-11-26T06:07:53.697169+00:00 app[web.1]: File "/app/myapp/views.py", line 126, in StartDrive 2021-11-26T06:07:53.697169+00:00 app[web.1]: pygame.mixer.init() 2021-11-26T06:07:53.697169+00:00 app[web.1]: pygame.error: Failed loading libasound.so.2: libasound.so.2: cannot open shared object file: No such file or directory 2021-11-26T06:07:53.697834+00:00 app[web.1]: 10.1.4.217 - - [25/Nov/2021:22:07:53 -0800] "GET /myapp/drive/ HTTP/1.1" 500 55987 "https://dontsleepapp.herokuapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0" 2021-11-26T06:07:53.701240+00:00 heroku[router]: at=info method=GET path="/myapp/drive/" host=dontsleepapp.herokuapp.com request_id=4f1d6ac0-8d16-4bd0-8ac9-381ed4758b91 fwd="73.93.42.18" dyno=web.1 connect=0ms service=66ms status=500 bytes=56254 protocol=https 

Answer: Your machine is missing libasound, install it with: sudo apt-get install libasound2
Question: Suppose I have a list like this, where numbers increase in different steps: [ 0, 4, 6, 8, 12, 15, 19, 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]  I want to return the index for the first element in the list where the increase is incremental (+1 step only). In this case, 23 is the first location from which point the increase becomes incremental, and its index would be 8, which is what I want as an output. What would be an elegant simple way to achieve this? This is what I have tried: >>> for (a,b) in zip(l, l[1:]): ... if b-a == 1: ... print(l.index(a)) ... break  UPDATE: In this particular setup, once the increase becomes incremental it will continue to stay that way. It is possible that the increase will never become incremental.

Answer: Solution 1: operator from operator import sub, indexOf L = [ 0, 4, 6, 8, 12, 15, 19, 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32] print(indexOf(map(sub, L[1:], L), 1)) # prints 8  Raises ValueError: sequence.index(x): x not in sequence if difference 1 never occurs, so might want to use try/except for that. Solution 2: bisect This one only takes O(log n) time, using the monotonicity of incrementalness (as you commented "once the increase becomes incremental it will continue to stay that way"). from bisect import bisect L = [ 0, 4, 6, 8, 12, 15, 19, 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32] class IsIncremental: def __getitem__(_, i): return L[i+1] - L[i] == 1 print(bisect(IsIncremental(), False, 0, len(L) - 1)) # prints 8  Prints len(L) - 1 if difference 1 never occurs. Btw... readability As PEP 8 says:  Never use the characters 'l' (lowercase letter el), [...] as single character variable names. In some fonts, these characters are indistinguishable from the numerals one and zero. When tempted to use 'l', use 'L' instead.
Question: I've been following this keras video classification tutorial where in the data preparation section, they load the frames of a video in the load_video function pretty generically, but what caught my eye was this line: frame = frame[:, :, [2, 1, 0]]  This is the first time I encounter this, most of the times you will just append the frame "as-is" to your list of frames, but here they change the order of the channels (if I'm not mistaken) from RGB to BGR, I couldn't find anything related to it in the web or their docs, can someone give me some insight for this decision?

Answer: From experience, the reason why the order can change is dependent on the framework you are using to load in images. OpenCV in particular orders the channels in BGR format because of mostly historical reasons that are now outdated. Because of this, we are unfortunately stuck with this design choice. Images in the regular RGB format can be seen with scikit-image, matplotlib and Pillow. In fact, if you look at the load_video function, it uses OpenCV to open up a video so the frames coming in are BGR format. Therefore, the swapping of channels is mandatory to get it to RGB format: def load_video(path, max_frames=0): cap = cv2.VideoCapture(path) frames = [] try: while True: ret, frame = cap.read() if not ret: break frame = crop_center(frame) frame = frame[:, :, [2, 1, 0]] frames.append(frame) if len(frames) == max_frames: break finally: cap.release() return np.array(frames)  You of course do not need to reverse the channels as a neural network will learn based on the input data that it was provided, but people tend to do this so that it's easy to debug images and not have to worry about continuously reversing the channels for display. Specifically, if a neural network was trained in BGR ordering, if you loaded in images in RGB format then the reversal of the channels needs to be done as that was how the image channels were represented in training. All in all, it depends on the framework but you need to keep this in mind when using a neural network after it has been trained. If the data was trained in BGR format, if your images are read in RGB format you'll need to reverse the channels prior to inference. In fact, this is a common bug when using networks! Be extremely diligent and understand how the image data was preprocessed for the network before using it.