eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So I have been trying to get it so there is a label at the end of each line giving the name of the country, then I can remove the legend. Have tried playing with transform_filter but no luck. I used data from here https://ourworldindata.org/coronavirus-source-data I cleaned and reshaped the data so it looks like this:-  index days date country value 0 1219 0 2020-03-26 Australia 11.0 1 1220 1 2020-03-27 Australia 13.0 2 1221 2 2020-03-28 Australia 13.0 3 1222 3 2020-03-29 Australia 14.0 4 1223 4 2020-03-30 Australia 16.0 5 1224 5 2020-03-31 Australia 19.0 6 1225 6 2020-04-01 Australia 20.0 7 1226 7 2020-04-02 Australia 21.0 8 1227 8 2020-04-03 Australia 23.0 9 1228 9 2020-04-04 Australia 30.0  import altair as alt countries_list = ['Australia', 'China', 'France', 'Germany', 'Iran', 'Italy','Japan', 'South Korea', 'Spain', 'United Kingdom', 'United States'] chart = alt.Chart(data_core_sub).mark_line().encode( alt.X('days:Q'), alt.Y('value:Q', scale=alt.Scale(type='log')), alt.Color('country:N', scale=alt.Scale(domain=countries_list,type='ordinal')), ) labels = alt.Chart(data_core_sub).mark_text().encode( alt.X('days:Q'), alt.Y('value:Q', scale=alt.Scale(type='log')), alt.Text('country'), alt.Color('country:N', legend=None, scale=alt.Scale(domain=countries_list,type='ordinal')), ).properties(title='COVID-19 total deaths', width=600) alt.layer(chart, labels).resolve_scale(color='independent')  This is the current mess that the chart is in.  How would I go about just showing the last 'country' name? EDIT Here is the result. I might look at adjusting some of the countries separately as adjusting as a group means that some of the labels are always badly positioned no matter what I do with the dx and dy alignment. 

Answer: You can do this by aggregating the x and y encodings. You want the text to be at the maximum x value, so you can use a 'max' aggregate in x. For the y-value, you want the y value associated with the max x-value, so you can use an {"argmax": "x"} aggregate. With a bit of adjustment of text alignment, the result looks like this: labels = alt.Chart(data_core_sub).mark_text(align='left', dx=3).encode( alt.X('days:Q', aggregate='max'), alt.Y('value:Q', aggregate={'argmax': 'days'}, scale=alt.Scale(type='log')), alt.Text('country'), alt.Color('country:N', legend=None, scale=alt.Scale(domain=countries_list,type='ordinal')), ).properties(title='COVID-19 total deaths', width=600)
Question: I am trying to run a basic flask app inside a docker container. The docker build works fine but when i try to test locally i get   127.0.0.1 didn't send any data error.   Dockerfile FROM tiangolo/uwsgi-nginx-flask:python3.6-alpine3.7 ENV LISTEN_PORT=5000 EXPOSE 5000 RUN pip install --upgrade pip WORKDIR /app ADD . /app CMD ["python3","main.py","--host=0.0.0.0"]  main.py import flask from flask import Flask, request import os app = Flask(__name__) @app.route('/') def this_works(): return "This works..." if __name__ == '__main__': app.run(debug=True)  The command to run the container i am using is : docker run -it --name dockertestapp1 --rm -p 5000:5000 dockertestapp1  Also command to build is : docker build --tag dockertestapp1 .  Could someone help please.

Answer: The issue is that you are passing --host parameter while not using the flask binary to bring up the application. Thus, you need to just take the parameter out of CMD in Dockerfile to your code. Working setup: Dockerfile: FROM tiangolo/uwsgi-nginx-flask:python3.6-alpine3.7 ENV LISTEN_PORT=5000 EXPOSE 5000 RUN pip install --upgrade pip WORKDIR /app ADD . /app CMD ["python3","main.py"]  and main.py import flask from flask import Flask, request import os app = Flask(__name__) @app.route('/') def this_works(): return "This works..." if __name__ == '__main__': app.run(host="0.0.0.0", debug=True)  The way you are building the image and bringing up the container is correct. I am adding the steps again for the answer to be whole: # build the image docker build --tag dockertestapp1 . # run the container docker run -it --name dockertestapp1 --rm -p 5000:5000 dockertestapp1
Question: I am trying to run an example for upscaling images from the following website: https://towardsdatascience.com/deep-learning-based-super-resolution-with-opencv-4fd736678066 This is the code I am using: import cv2 from cv2 import dnn_superres # Create an SR object sr = dnn_superres.DnnSuperResImpl_create() # Read image image = cv2.imread('butterfly.png') # Read the desired model path = "EDSR_x3.pb" sr.readModel(path) # Set the desired model and scale to get correct pre- and post-processing sr.setModel("edsr", 3) # Upscale the image result = sr.upsample(image) # Save the image cv2.imwrite("./upscaled.png", result)  I have downloaded the already trained model from the website, called "EDSR_x3.pb" and when I run the code I get the following error: Traceback (most recent call last): File "upscale.py", line 2, in <module> from cv2 import dnn_superres ImportError: cannot import name 'dnn_superres'  I now it seems like there is no such method or class, but I have already installed opencv and the contrib modules. Why do I get this error?

Answer: I had the same problem with Python 3.6.9 and opencv 4.2.0, but after the upgrade to 4.3.0, the problem disappeared. If you have no problem upgrading the version, try 4.3.0.
Question: I'm learning pytorch and I'm wondering what does the padding_idx attribute do in torch.nn.Embedding(n1, d1, padding_idx=0)? I have looked everywhere and couldn't find something I can get. Can you show example to illustrate this?

Answer: As per the docs, padding_idx pads the output with the embedding vector at padding_idx (initialized to zeros) whenever it encounters the index.  What this means is that wherever you have an item equal to padding_idx, the output of the embedding layer at that index will be all zeros. Here is an example: Let us say you have word embeddings of 1000 words, each 50-dimensional ie num_embeddingss=1000, embedding_dim=50. Then torch.nn.Embedding works like a lookup table (lookup table is trainable though): emb_layer = torch.nn.Embedding(1000,50) x = torch.LongTensor([[1,2,4,5],[4,3,2,9]]) y = emb_layer(x)  y will be a tensor of shape 2x4x50. I hope this part is clear to you. Now if I specify padding_idx=2, ie emb_layer = torch.nn.Embedding(1000,50, padding_idx=2) x = torch.LongTensor([[1,2,4,5],[4,3,2,9]]) y = emb_layer(x)  then output will still be 2x4x50 but the 50-dim vector at (1,2) and (2,3) will be all zeros since x[1,2] and x[2,3] values are 2 which is equal to the padding_idx. You can think of it as 3rd word in the lookup table (since lookup table would be 0-indexed) is not being used for training.
Question: Given two input i. original image & ii. mask image, what's the best way to remove to the background from the original image.  Original Image  Mask Image  The final output would contain just the dog without the background and look transparent. I have seen the mask images are also created with OpenCV. Is there a way to just the existing mask image and generate the output image? Update I tried this  import cv2 # opencv loads the image in BGR, convert it to RGB img = cv2.imread("originalImage.png") mask = cv2.imread("maskImage.png") final = cv2.bitwise_and(img, mask) cv2.imwrite("final.png", final)  Final Image  Is there a way to set the background to be transparent?

Answer: You can create a transparent image by creating a 4-channel BGRA image and copying the first 3 channels from the original image and setting the alpha channel using the mask image. transparent = np.zeros((img.shape[0], img.shape[1], 4), dtype=np.uint8) transparent[:,:,0:3] = img transparent[:, :, 3] = mask
Question: Using fastapi, I can't figure out how to send multiple files as a response. For example, to send a single file, I'll use something like this from fastapi import FastAPI, Response app = FastAPI() @app.get("/image_from_id/") async def image_from_id(image_id: int): # Get image from the database img = ... return Response(content=img, media_type="application/png")  However, I'm not sure what it looks like to send a list of images. Ideally, I'd like to do something like this: @app.get("/images_from_ids/") async def image_from_id(image_ids: List[int]): # Get a list of images from the database images = ... return Response(content=images, media_type="multipart/form-data")  However, this returns the error  def render(self, content: typing.Any) -> bytes: if content is None: return b"" if isinstance(content, bytes): return content > return content.encode(self.charset) E AttributeError: 'list' object has no attribute 'encode' 

Answer: Zipping is the best option that will have same results on all browsers. you can zip files dynamically. import os import zipfile import StringIO def zipfiles(filenames): zip_subdir = "archive" zip_filename = "%s.zip" % zip_subdir # Open StringIO to grab in-memory ZIP contents s = StringIO.StringIO() # The zip compressor zf = zipfile.ZipFile(s, "w") for fpath in filenames: # Calculate path for file in zip fdir, fname = os.path.split(fpath) zip_path = os.path.join(zip_subdir, fname) # Add file, at correct path zf.write(fpath, zip_path) # Must close zip for all contents to be written zf.close() # Grab ZIP file from in-memory, make response with correct MIME-type resp = Response(s.getvalue(), mimetype = "application/x-zip-compressed") # ..and correct content-disposition resp['Content-Disposition'] = 'attachment; filename=%s' % zip_filename return resp @app.get("/image_from_id/") async def image_from_id(image_id: int): # Get image from the database img = ... return zipfiles(img)  As alternative you can use base64 encoding to embed an (very small) image into json response. but i don't recommend it. You can also use MIME/multipart but keep in mind that i was created for email messages and/or POST transmission to the HTTP server. It was never intended to be received and parsed on the client side of a HTTP transaction. Some browsers support it, some others don't. (so i think you shouldn't use this either)
Question: I'm learning about decorators and came across an example where the decorator took an argument. This was a little confusing for me though, because I learned that (note: the examples from this question are mostly from this article): def my_decorator(func): def inner(*args, **kwargs): print('Before function runs') func(*args, **kwargs) print('After function ran') return inner @my_decorator def foo(thing_to_print): print(thing_to_print) foo('Hello') # Returns: # Before function runs # Hello # After function ran  was equivalent to  foo = my_wrapper(foo)  So, it doesn't make sense to me how something could take an argument, to better explain, here is a decorator example that takes an argument: def repeat(num_times): def decorator_repeat(func): @functools.wraps(func) def wrapper_repeat(*args, **kwargs): for _ in range(num_times): value = func(*args, **kwargs) return value return wrapper_repeat return decorator_repeat @repeat(num_times=4) def greet(name): print(f"Hello {name}") greet('Bob') # Returns: # Hello Bob # Hello Bob # Hello Bob # Hello Bob  So when I see this I am thinking: greet = repeat(greet, num_times=4)  I know that that can't be right because num_times is the only argument that should be getting passed. So what is the correct equivalent to @repeat(num_times=4) without the "@-symbol-syntax"? Thanks!

Answer: That article is the same one that tought me everthing I know about decorators! It's brilliant. In regards to what the non @ symbol syntax looks like: You can imagine the actual decorator function is decorator_repeat(func), the function within repeat(num_times=4). @repeat(num_times=4) returns a decorator which is essentially @decorator_repeat except @decorator_repeat now has access to a variable num_times. Further down the page in the article it shows how to make these arguments optional which may help further clarify it for you.
Question: I have an input text which looks like this: word77 text text bla66 word78 text bla67 text bla68 word79 text bla69 word80 text bla77 word81 text bla78 word92 text bla79 word99  I have to renumber word and bla from 1, in each line. I can renumber the whole input which looks like this: word1 text text bla1 word2 text bla2 text bla3 word3 text bla4 word4 text bla5 word5 text bla6 word6 text bla7 word7  The code for the above: import re def replace(m): global i; i+=1; return str(i); fp = open('input.txt', 'r').read() i = 0 fp = re.sub(r'(?<=word)(\d+)', replace, fp) i = 0 fp = re.sub(r'(?<=bla)(\d+)', replace, fp) #open('sample.txt', 'wb').write(fp) print fp  Ideally, the result should look like this: word1 text text bla1 word2 text bla2 text bla1 word1 text bla2 word2 text bla1 word2 text bla3 word3 text bla4 word4 

Answer: You operate on the whole file at once (fp.read()) - you need to do it line-wise: with open("input.txt","w") as f: f.write("""word77 text text bla66 word78 text bla67 text bla68 word79 text bla69 word80 text bla77 word81 text bla78 word92 text bla79 word99""") import re i = 0 def replace(m): global i i+=1 return str(i) with open('input.txt') as fp, open("output.txt","w") as out: # read only one line of the file and apply the transformations for line in fp: i = 0 l = re.sub(r'(?<=word)(\d+)', replace, line) i = 0 l = re.sub(r'(?<=bla)(\d+)', replace, l) out.write(l) with open("output.txt") as f: print(f.read())  Output: word1 text text bla1 word2 text bla2 text bla1 word1 text bla2 word2 text bla1 word1 text bla2 word2 text bla3 word3
Question: I updated my system (Ubuntu 18.04) from Python 3.6 to Python 3.8, and reset the defaults so that python3 now points to Python 3.8 (and not 3.6). However, since then, the terminal has refused to open using Ctrl + Alt + T, and other obvious methods such as clicking on the icon itself.  When I run gnome-terminal - I get the following: usernew@HP:/usr/lib/python3/dist-packages/gi$ gnome-terminal Traceback (most recent call last): File "/usr/bin/gnome-terminal", line 9, in <module> from gi.repository import GLib, Gio File "/usr/lib/python3/dist-packages/gi/__init__.py", line 42, in <module> from . import _gi ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (/usr/lib/python3/dist-packages/gi/__init__.py)  I don't know what this means but I guess it definitely points to the fact that something went wrong during the update. I understand that there are other existing threads on similar issues, but most of them were about updating from Python2 to Python3, so I'm not sure if they're relevant.  Could someone help, please? Important Update: So, after reading this answer - I changed the gnome-terminal script's first line to #!/usr/bin/python3.6 instead of #!/usr/bin/python3.8 - and that solves the problem. Also, when I type python3 in the terminal, I'm greeted with Python 3.8.2, as desired.  The question remains - Why did this work? What was the actual problem? An explanation would help, so I really know what I'm doing. Thanks!

Answer: You shouldn't change the symlink /usr/bin/python3 since a bunch of Ubuntu components depend on it, and Ubuntu-specific Python libraries like gi are built only for the Python build shipped with Ubuntu, which is version 3.6 on 18.04. See Gnome terminal will not start on Ask Ubuntu (though note that it's about Ubuntu 16.04 which uses Python 3.5). So the best way to fix it is to revert the symlink: sudo ln -sf python3.6 /usr/bin/python3  As for setting Python 3.8 as the default, you could put an alias in your bashrc: alias python3=python3.8  But this will only affect the shell for your user. In scripts for example if you want to use Python 3.8 you'll have to write it, i.e. #!/usr/bin/env python3.8
Question: I could not install MetaTrader5 by: pip install MetaTrader5 I got the following error: ERROR: Could not find a version that satisfies the requirement MetaTrader5 (from versions: none) ERROR: No matching distribution found for MetaTrader5 Knowing that I am on MAC laptop and I have Python 3.7.6 Thanks in advance to provide me the solution

Answer: MetaTrader5 provides a lot of binary wheels but only for w32 and w64. No Linux, no MacOS and no source code. It seems the software is Windows-only. Their site recommends to use one of the w32/w64 emulators on MacOS.
Question: While learning python I came across a line of code which will figure out the numbers of letters.  dummy='lorem ipsum dolor emet...' letternum={} for each_letter in dummy: letternum[each_letter.lower()]=letternum.get(each_letter,0)+1 print(letternum)  Now, My question is -in the 4th line of code inletternum.get(each_letter,0)+1 why there is ,0)+1and why is it used for. Pls describe.

Answer: The get method on a dictionary is documented here: https://docs.python.org/3/library/stdtypes.html#dict.get  get(key[, default]) Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.  So this explains the 0 - it's a default value to use when letternum doesn't contain the given letter. So we have letternum.get(each_letter, 0) - this expression finds the value stored in the letternum dictionary for the currently considered letter. If there is no value stored, it evaluates to 0 instead. Then we add one to this number: letternum.get(each_letter, 0) + 1 Finally we stored it back into the letternum dictionary, although this time converting the letter to lowercase: letternum[each_letter.lower()] = letternum.get(each_letter, 0) + 1 It seems this might be a mistake. We probably want to update the same item we just looked up, but if each_letter is upper-case that's not true.
Question: I converted some audio files to spectrograms and saved them to files using the following code: import os from matplotlib import pyplot as plt import librosa import librosa.display import IPython.display as ipd audio_fpath = "./audios/" spectrograms_path = "./spectrograms/" audio_clips = os.listdir(audio_fpath) def generate_spectrogram(x, sr, save_name): X = librosa.stft(x) Xdb = librosa.amplitude_to_db(abs(X)) fig = plt.figure(figsize=(20, 20), dpi=1000, frameon=False) ax = fig.add_axes([0, 0, 1, 1], frameon=False) ax.axis('off') librosa.display.specshow(Xdb, sr=sr, cmap='gray', x_axis='time', y_axis='hz') plt.savefig(save_name, quality=100, bbox_inches=0, pad_inches=0) librosa.cache.clear() for i in audio_clips: audio_fpath = "./audios/" spectrograms_path = "./spectrograms/" audio_length = librosa.get_duration(filename=audio_fpath + i) j=60 while j < audio_length: x, sr = librosa.load(audio_fpath + i, offset=j-60, duration=60) save_name = spectrograms_path + i + str(j) + ".jpg" generate_spectrogram(x, sr, save_name) j += 60 if j >= audio_length: j = audio_length x, sr = librosa.load(audio_fpath + i, offset=j-60, duration=60) save_name = spectrograms_path + i + str(j) + ".jpg" generate_spectrogram(x, sr, save_name)  I wanted to keep the most detail and quality from the audios, so that i could turn them back to audio without too much loss (They are 80MB each). Is it possible to turn them back to audio files? How can I do it?  I tried using librosa.feature.inverse.mel_to_audio, but it didn't work, and I don't think it applies. I now have 1300 spectrogram files and want to train a Generative Adversarial Network with them, so that I can generate new audios, but I don't want to do it if i wont be able to listen to the results later.

Answer: Yes, it is possible to recover most of the signal and estimate the phase with e.g. Griffin-Lim Algorithm (GLA). Its "fast" implementation for Python can be found in librosa. Here's how you can use it: import numpy as np import librosa y, sr = librosa.load(librosa.util.example_audio_file(), duration=10) S = np.abs(librosa.stft(y)) y_inv = librosa.griffinlim(S)  And that's how the original and reconstruction look like:  The algorithm by default randomly initialises the phases and then iterates forward and inverse STFT operations to estimate the phases. Looking at your code, to reconstruct the signal, you'd just need to do: import numpy as np X_inv = librosa.griffinlim(np.abs(X))  It's just an example of course. As pointed out by @PaulR, in your case you'd need to load the data from jpeg (which is lossy!) and then apply inverse transform to amplitude_to_db first. The algorithm, especially the phase estimation, can be further improved thanks to advances in artificial neural networks. Here is one paper that discusses some enhancements.
Question: When opening a .hdf5 file, one can explore the levels, keys and names of the file in different ways. I wonder if there is a way or a function that displays all the available paths to explore in the .hdf5. Ultimately showing the whole tree.

Answer: Try using nexuformat package to list the structure of the hdf5 file. Install by pip install nexusformat Code  import nexusformat.nexus as nx f = nx.nxload(‘myhdf5file.hdf5’) print(f.tree)  This should print the entire structure of the file. For more on that see this thread. Examples can be found here
Question: I use scatter to plot some points. For example: import numpy as np import matplotlib.pyplot as plt import matplotlib.animation as animation fig, ax = plt.subplots() ax.scatter([1,2, 1.5], [2, 1, 1.5]) plt.show()  Now I also want a circle with radius 0.5 around point [1.5, 1.5] in the plot. How do I do that? I know that there are edgecolors, so that I could just set them to 'none' and then to some color. But those circles then have no radius of 0.5.

Answer: To make a circle around point, you can use plt.Circle as shown below: import numpy as np import matplotlib.pyplot as plt import matplotlib.animation as animation fig, ax = plt.subplots() ax.scatter([1,2, 1.5], [2, 1, 1.5]) cir = plt.Circle((1.5, 1.5), 0.07, color='r',fill=False) ax.set_aspect('equal', adjustable='datalim') ax.add_patch(cir) plt.show()  I hope this solved your problem. And please let me know further.
Question: I need to find consecutive (non-overlapping) repetitions of a substring in a string. I can count them but not consecutive. For instance: string = "AASDASDDAAAAAAAAERQREQREQRAAAAREWQRWERAAA" substring = "AA"  here, "AA" is repeated one time at the beginning of the string, then 4 times, then 2 times, etc. I should select the biggest one, in this example - 4 times. How can I do that?

Answer: Regular expressions shine when searching through strings. Here you can find all groups of one or more AA with (?:AA)+ the (?: simply tells the engine to interpret the parentheses for grouping only. Once you have the groups you can use max() to find the longest based on length (len()).  import re s = "AASDASDDAAAAAAAAERQREQREQRAAAAREWQRWERAAA" groups = re.findall(r'(?:AA)+', s) print(groups) # ['AA', 'AAAAAAAA', 'AAAA', 'AA'] largest = max(groups, key=len) print(len(largest) // 2) # 4
Question: If I try to overwrite an existing blob: blob_client = BlobClient.from_connection_string(connection_string, container_name, blob_name) blob_client.upload_blob('Some text')  I get a ResourceExistsError. I can check if the blob exists, delete it, and then upload it: try: blob_client.get_blob_properties() blob_client.delete_blob() except ResourceNotFoundError: pass blob_client.upload_blob('Some text')  Taking into account both what the python azure blob storage API has available as well as idiomatic python style, is there a better way to overwrite the contents of an existing blob? I was expecting there to be some sort of overwrite parameter that could be optionally set to true in the upload_blob method, but it doesn't appear to exist.

Answer: From this issue it seems that you can add overwrite=True to upload_blob and it will work.