eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Consider the following typical python project structure: fooproject - main.py - src/ - test/ - logs/ - Dockerfile - .dockerignore - README.md  The .dockerfile should prevent test/ and logs/ directories from being included in the docker image. test/ logs/  Contents of Dockerfile are FROM ubuntu16.04 COPY . /app/ WORKDIR /app USER root RUN pip install -r requirements.txt ENTRYPOINT ["main.py"]  However, when running through PyCharm's automatic docker integration, the test and logs directories are both copied into the container. The PyCharm run command ends up as follows:  7eb643d9785b:python -u /opt/project/main.py  I've tried a few things to no avail, such as making a duplicate copy of .dockerignore at the directory above the rest of the app. For example: COPY . /app/ COPY .dockerignore .dockerignore WORKDIR /app  Wondering if it's possible that PyCharm's /opt/project is somehow interfering? So where exactly should .dockerignore be in a project like this? Update I shared the output of docker container inspect with BMitch, and he was able to find the solution. PyCharm was automatically mounting a volume in the container settings run config 

Answer: The .dockerignore is used to control what files are included in the build context. This impacts the COPY and ADD commands in the Dockerfile, and ultimately the resulting image. When you run that image with a volume mount, e.g.:  { "Type": "bind", "Source": "/home/adam/Desktop/Dev/ec2-data-analysis/grimlock", "Destination": "/opt/project", "Mode": "rw", "RW": true, "Propagation": "rprivate" },  That volume mount overrides the contents of the image for that container. All access to the path will go to your desktop directory rather than the image contents, and Linux bind mounts do not have a concept of the .dockerignore file. When you run this image without the volume mount, you should see different behavior.  For anyone coming across this question in the future, the .dockerignore needs to be in the root of your build context. The build context is the directory you pass at the end of the build command, often a . or the current directory. If you enable BuildKit, it will first check for a Dockerfile.dockerignore where your Dockerfile path/name could be changed. And to test your .dockerignore, see this answer.
Question: How likely is it for a collision to occur with tokens generated using Python's secrets library (https://docs.python.org/3/library/secrets.html)? There doesn't seem to be any mention of their uniqueness.

Answer: The purpose of the secrets module, is for sourcing secret data, i.e. information that cannot be predicted or reverse engineered  The secrets module provides access to the most secure source of randomness that your operating system provides.  Typically, the OS will use several entropy sources to generate bits. e.g. process id, thread id, mouse / keyboard input timings, CPU counters, system time, etc. So long as the amount of entropy is sufficiently large for the amount of bits being generated (and again the OS uses many sources and accumulates entropy constantly), we should expect a uniform distribution of all values. So if you generate a 32 bit key, you should expect to see each of the 4294967296 values with similar frequency. To estimate how long before we expect a collision, we are essentially looking at the Birthday problem. In general, if the values are uniformly distributed, and the number of values is n, we should expect a collision to occur after generating about sqrt(n) values (though in reality it's a bit more). You can verify this with a quick benchmark program import secrets def find_collision(b): tokens = set() while True: token = secrets.token_bytes(b) if token in tokens: return len(tokens) tokens.add(token) b = 4 samples = 100 l = [find_collision(b) for i in range(samples)] avg = sum(l)/len(l) print(f'n = {2**(b*8)}, sqrt(n) = {2**(b*8/2)}') print(f'on average, with a {b} byte token, a collision occurs after {avg} values')  n = 4294967296, sqrt(n) = 65536.0 on average, with a 4 byte token, a collision occurs after 75797.78 values  secrets makes no aims or claims about uniqueness, as the goal is to generate high entropy, random bytes that should be impossible to predict or reverse engineer. To add constraints to the module to prevent duplicates would inherently make it more predictable. In addition, secrets feeds you these bytes in a stream under the assumption that you take what you need, and use it how you like, so preventing duplicates doesn't make much sense as an upstream responsibility. This is in contrast to something like a UUID, which has a fixed width and is intended to be used as a portable identifier and recognized datatype.
Question: I am using Python 3.6 and flask. I used flask-mysqldb to connect to MySQL, but whenever I try to run mypy on my program I get this error:  Skipping analyzing 'flask_mysqldb': found module but no type hints or library stubs.  I tried running mypy with the flags ignore-missing-imports or follow-imports=skip. Then I was not getting the error. Why do I get this error? How can I fix this without adding any additional flags?

Answer: You are getting this error because mypy is not designed to try type checking every single module you try importing. This is mainly for three reasons:  The module you're trying to import could be written in a way that it fails to type check. For example, if the module does something like my_list = [] in the global scope, mypy will ask for a type hint since it doesn't know what this list is supposed to contain. These sorts of errors are out of the control of people using the libraries, and so it would be annoying and disruptive to potentially spam them everywhere. Even if the library module you're trying to import type checks cleanly, if it's not using type hints, you won't gain much benefit from trying to use it with mypy. If the library is dynamically typed, mypy could silently accept code that actually does not type check at runtime. This can be surprising to people/silently doing the wrong thing is generally a bad idea. Instead, you get an explicit warning. Not all modules are written in Python -- some modules are actually C extensions. Mypy cannot analyze these modules and so must implement some mechanism for ignoring modules.  If you do not get this error, this means one of five things:  The type hints for your library already exist in Typeshed, which comes pre-bundled with mypy. Typeshed mostly contains type hints for the standard library and a handful of popular 3rd party libraries. The library is already using type hints and declared that it wants to be analyzed and type checked by mypy. This is done by including a special py.typed file within the package which makes it PEP 561 compatible. You've installed a 3rd party "stubs-only" package. This package can be installed alongside the library and lets people provide type hints without having to modify the library itself. For example, the django-stubs package contains type hints for the Django library, which is not PEP 561 compatible. You've configured mypy to use your own custom stubs. That is, you've basically created your own local "stubs-only" package and told mypy to use it. You've decided to suppress the error and live with the fact that the library is dynamically typed for now -- for example, by using the command line flags you found or by adding a # type: ignore to the import.  For more details on how to deal with this error, see the mypy docs on dealing with missing type hints from 3rd party libraries.
Question: What are the caveats (if any) of using a class that inherits from both str and Enum? This was listed as a possible way to solve the problem of Serialising an Enum member to JSON from enum import Enum class LogLevel(str, Enum): DEBUG = 'DEBUG' INFO = 'INFO'  Of course the point is to use this class as an enum, with all its advantages 

Answer: When inheriting from str, or any other type, the resulting enum members are also that type. This means:  they have all the methods of that type they can be used as that type and, most importantly, they will compare with other instances of that type  That last point is the most important: because LogLevel.DEBUG is a str it will compare with other strings -- which is good -- but will also compare with other str-based Enums -- which could be bad. Info regarding subclassing enum from the documentation
Question: I'm making a Discord bot in Python using discord.py. I'd like to set/modify a global variable from an async thread. message = "" @bot.command() async def test(ctx, msg): message = msg  However this doesn't work. How can I achieve something that does this?

Answer: As I said in the comment you have to use the keyword global in the functions wherever you are modifying the global variable. If you are just reading it in function than you don't need it.  message = "" @bot.command() async def test(ctx, msg): global message message = msg
Question: I have a Series of strings, the series looks like; Series "1, 2, 6, 7, 6" "1, 3, 7, 9, 9" "1, 1, 3, 5, 6" "1, 2, 7, 7, 8" "1, 4, 6, 8, 9" "1"  I want to remove all elements apart from the first and last, so the output would look like; Series "1, 6" "1, 9" "1, 6" "1, 8" "1, 9" "1"  To use Split(), do I need to loop over each element in the series? I've tried this but can't get the output I'm looking for.

Answer: You can use split and rsplit to get the various parts: result = [f"{x.split(',', 1)[0]},{x.rsplit(',', 1)[1]}" if x.find(',') > 0 else x for x in strings]  If strings is a pd.Series object then you can convert it back to a series: result = pd.Series(result, index=strings.index)
Question: just a quick question, where I couldn't find anything helpful in the plt.errorbar documentation I want to plot values with error bars: import matplotlib.pyplot as plt plt.errorbar(1, 0.25, yerr=0.1, uplims=True, lolims=True, fmt='o') plt.show()   But I would like to have error bars with a simple horizontal line instead of arrows at the ends. But there is no "capmarker" or similar option in the plt.errorbar() function

Answer: Remove the uplims=True and lolims=True; both limits are plotted by default, without any ending arrows: import matplotlib.pyplot as plt plt.errorbar(1, 0.25, yerr=0.1, fmt='o') plt.show()   EDIT: Increase the capsize to add caps to the end of the error bars, and increase the capthick to make the caps thicker: plt.errorbar(1, 0.25, yerr=0.1, fmt='o', capsize=3)   plt.errorbar(1, 0.25, yerr=0.1, fmt='o', capsize=3, capthick=3)
Question: I would like to install a few packages from a github project and one of the dependencies is python@2.  Prior to Jan 1 2020, it was possible to install python@2 using Homebrew: $ brew install python@2  However, Python 2 support has ended from Homebrew. Is there anyway to install python@2 on a Mac now that Python 2 support has ended?  Until the code in this project is ported to Python 3, unfortunately I'm stuck with getting it to work with Python 2 (and dependencies which use Python 2), which is the reason I would like to install python@2 as a temporary solution.

Answer: I was able to find the exact same question -- Brew - reinstalling python@2. Unfortunately I can't raise the duplicate flag again so I'll duplicate the answer: brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86a44a0a552c673a05f11018459c9f5faae3becc/Formula/python@2.rb
Question: I tried generating and combining two unimodal distributions but think there's something wrong in my code. N=400 mu, sigma = 100, 5 mu2, sigma2 = 10, 40 X1 = np.random.normal(mu, sigma, N) X2 = np.random.normal(mu2, sigma2, N) w = np.random.normal(0.5, 1, N) X = w*X1 + (1-w)*X2 X = X.reshape(-1,2)  When I plot X I don't get a bimodal distribution

Answer: It's unclear where your problem is; it's also unclear what the purpose of the variable w is, and it's unclear how you judge you get an incorrect result, since we don't see the plot code, or any other code to confirm or reject a binomial distribution. That is, your example is too incomplete to exactly answer your question. But I can make an educated guess. If I do the following below: import numpy as np import matplotlib.pyplot as plt N=400 mu, sigma = 100, 5 mu2, sigma2 = 10, 40 X1 = np.random.normal(mu, sigma, N) X2 = np.random.normal(mu2, sigma2, N) X = np.concatenate([X1, X2]) plt.hist(X)  and that yields the following figure:
Question: Say I have a pandas DataFrame with four columns: A,B,C,D. my_df = pd.DataFrame({'A': [0,1,4,9], 'B': [1,7,5,7],'C':[1,1,1,1],'D':[2,2,2,2]})  I also have a list of tuples: my_tuples = [(0,1),(4,5),(9,9)]  I want to keep only the rows of the dataframe where the value of (my_df['A'],my_df['B']) is equal to one of the tuples in my_tuples. In this example, this would be row#0 and row#2. Is there a good way to do this? I'd appreciate any help.

Answer: Use DataFrame.merge with DataFrame created by tuples, there is no on parameter for default interecton of all columns in both DataFrames, here A and B: df = my_df.merge(pd.DataFrame(my_tuples, columns=['A','B'])) print (df) A B C D 0 0 1 1 2 1 4 5 1 2  Or: df = my_df[my_df.set_index(['A','B']).index.isin(my_tuples)] print (df) A B C D 0 0 1 1 2 2 4 5 1 2
Question: I come from Java where Exceptions and Errors are quite different things and they both derive from something called Throwable. In Java normally you should never try to catch an Error. In Python though it seems the distinction is blurred. So far after reading some docs and checking the hierarchy I have the following questions:  There are syntax errors which of course cause your program not to be able to start at all. Right?  "Errors detected during execution are called exceptions and are not unconditionally fatal" (per the tutorial). What does "fatal" mean here? Also, some objects like AttributeError are (by the above definition) actually exceptions even though they contain Error in their names, is that conclusion correct?  Some classes derive from Exception but contain Error in their name. Isn't this confusing? But even so it means that Error in the name is in no way special, it's still an Exception. Or not... ?  "All built-in, non-system-exiting exceptions are derived from [Exception]" (quote from here) So which ones are system-exiting exceptions and which ones are not? It is not immediately clear. All user-defined exceptions should also be derived from Exception. So basically as a beginner do I need to worry about anything else but Exception? Seems like not.  Warnings also derive from Exception. So are warnings fatal or system-exiting or none of these?  Where does the AssertionError fit into all of this? Is it fatal or system exiting?  How does one know or specify that some Exception class represents fatal or system-exiting exception?  

Answer:  Yes. SyntaxError isn't catchable except in cases of dynamically executed code (via eval/exec), because it occurs before the code is actually running. "Fatal" means "program dies regardless of what the code says"; that doesn't happen with exceptions in Python, they're all catchable. os._exit can forcibly kill the process, but it does so by bypassing the exception mechanism. There is no difference between exceptions and errors, so the nomenclature doesn't matter. System-exiting exceptions derive from BaseException, but not Exception. But they can be caught just like any other exception Warnings behave differently based on the warnings filter, and deriving from Exception means they're not in the "system-exiting" category AssertionError is just another Exception child class, so it's not "system exiting". It's just tied to the assert statement, which has special semantics. Things deriving from BaseException but not Exception (e.g. SystemExit, KeyboardInterrupt) are "not reasonable to catch" (or if you do catch them, it should almost always be to log/perform cleanup and rethrow them), everything else (derived from Exception as well) is "conditionally reasonable to catch". There is no other distinction.  To be clear, "system-exiting" is just a way of saying "things which except Exception: won't catch"; if no except blocks are involved, all exceptions (aside from warnings, which as noted, behave differently based on the warnings filter) are "system-exiting".
Question: Consider: text = "abcdef" pattern = "(b|e)cd(b|e)" repl = [r"\1bla\2", r"\1blabla\2"] text = re.sub(pattern, lambda m: random.choice(repl), text)  I want to replace matches randomly with entries of a list repl. But when using lambda m: random.choice(repl) as a callback, it doesn't replace \1, \2 etc. with its captures any more, returning "\1bla\2" as plain text. I've tried to look up re.py on how they do it internally, so I might be able to call the same internal function, but it doesn't seem trivial. The example above returns a\1bla\2f or a\1blabla\2f while abblaef or abblablaef are valid options in my case. Note that I'm using a function, because, in case of several matches like text = "abcdef abcdef", it should randomly choose a replacement from repl for every match – instead of using the same replacement for all matches.

Answer: If you pass a function you lose the automatic escaping of backreferences. You just get the match object and have to do the work. So you could: Pick a string in the regex rather than passing a function: text = "abcdef" pattern = "(b|e)cd(b|e)" repl = [r"\1bla\2", r"\1blabla\2"] re.sub(pattern, random.choice(repl), text) # 'abblaef' or 'abblablaef'  Or write a function that processes the match object and allows more complex processing. You can take advantage of expand to use back references: text = "abcdef abcdef" pattern = "(b|e)cd(b|e)" def repl(m): repl = [r"\1bla\2", r"\1blabla\2"] return m.expand(random.choice(repl)) re.sub(pattern, repl, text) # 'abblaef abblablaef' and variations  You can, or course, put that function into a lambda: repl = [r"\1bla\2", r"\1blabla\2"] re.sub(pattern, lambda m: m.expand(random.choice(repl)), text)
Question: I am trying to convert a pipe-delimited text file to a CSV file, and then iterate through and print the CSV file. Here is my code:  with open("...somefile.txt", "r") as text_file: text_reader = csv.reader(text_file, delimiter='|') with open("...somefile.csv", 'w') as csv_file: csv_writer = csv.writer(csv_file, delimiter=',') csv_writer.writerows(text_reader) with open (csv_file, 'r') as f: reader = csv.reader (f, delimiter=',') for row in reader: print(row)  However, I am getting this error message : ----> 9 with open (csv_file, 'r') as f: 10 reader = csv.reader (f, delimiter=',') 11 for row in reader: TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper  Can anyone explain what this means? Also, if I were to make this into a function, how could I take in a file name as input and then change the file to add a .csv extension wen converting to a csv file? Thanks

Answer: You're passing open an already opened file, rather than the path of the file you created. Replace: with open (csv_file, 'r') as f:  with  with open ("...somefile.csv", 'r') as f:  To change the extension in a function: import pathlib def txt_to_csv(fname): new_name = f'{Path(fname).stem}.csv' with open(fname, "r") as text_file: text_reader = csv.reader(text_file, delimiter='|') with open(new_name, 'w') as csv_file: csv_writer = csv.writer(csv_file, delimiter=',') csv_writer.writerows(text_reader) with open (new_name, 'r') as f: reader = csv.reader (f, delimiter=',') for row in reader: print(row)
Question: I'm new on python. I want to learn how to parallel processing in python. I saw the following example:  import multiprocessing as mp np.random.RandomState(100) arr = np.random.randint(0, 10, size=[20, 5]) data = arr.tolist() def howmany_within_range_rowonly(row, minimum=4, maximum=8): count = 0 for n in row: if minimum <= n <= maximum: count = count + 1 return count pool = mp.Pool(mp.cpu_count()) results = pool.map(howmany_within_range_rowonly, [row for row in data]) pool.close() print(results[:10])   but when I run it, this error happened: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.  What should I do?

Answer: If you place everything in global scope inside this if __name__ == "__main__" block as follows, you should find that your program behaves as you expect: def howmany_within_range_rowonly(row, minimum=4, maximum=8): count = 0 for n in row: if minimum <= n <= maximum: count = count + 1 return count if __name__ == "__main__": np.random.RandomState(100) arr = np.random.randint(0, 10, size=[20, 5]) data = arr.tolist() pool = mp.Pool(mp.cpu_count()) results = pool.map(howmany_within_range_rowonly, [row for row in data]) pool.close() print(results[:10])  Without this protection, if your current module was imported from a different module, your multiprocessing code would be executed. This could occur within a non-main process spawned in another Pool and spawning processes from sub-processes is not allowed, hence we protect against this problem.
Question: How can I change the colour of the plotly express bar-chart to green in the following code? import plotly.express as px import pandas as pd # prepare the dataframe df = pd.DataFrame(dict( x=[1, 2, 3], y=[1, 3, 2] )) # prepare the layout title = "A Bar Chart from Plotly Express" fig = px.bar(df, x='x', y='y', # data from df columns color= pd.Series('green', index=range(len(df))), # does not work title=title, labels={'x': 'Some X', 'y':'Some Y'}) fig.show() 

Answer: It can be done: - with color_discrete_sequence =['green']*3, or, - with fig.update_traces(marker_color='green') after the bar is instantiated. Ref: Community response
Question: My code sometimes produces a list of nan's op_list = [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]. I want to know if all elements are nans.  My code and present output:   op_list = [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan] print(np.isnan(op_list)) array([ True, True, True, True, True, True, True, True, True, True])  My expected output:   True 

Answer: You need all: np.isnan(op_list).all() # True  For a solution using lists you can do: all(i != i for i in op_list) # True
Question: I have a project of opencv where on the frame I am displaying some text using cv2.putText(). Currently it looks like below:  As you can see on the top left corner, the text is present but its not clearly visible. Is it possible to make background black so that the text will then appear good. Something like below image:  Even if the black background covers till right side of the frame, that is also fine. Below is the code I am using for putting text on frame: cv2.putText(frame, "Data: N/A", (5, 30), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0, 0, 255), 1) cv2.putText(frame, "Room: C1", (5, 60), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0, 0, 255), 1)  Is there any prebuilt method/library available in opencv which can do this. Can anyone please suggest a good way?

Answer: There's no prebuilt method but a simple appraoch is to use cv2.rectangle + cv2.putText. All you need to do is to draw the black rectangle on the image followed by placing the text. You can adjust the x,y,w,h parameters depending on how large/small you want the rectangle. Here's an example: Input image:  Result:  import cv2 import numpy as np # Load image, define rectangle bounds image = cv2.imread('1.jpg') x,y,w,h = 0,0,175,75 # Draw black background rectangle cv2.rectangle(image, (x, x), (x + w, y + h), (0,0,0), -1) # Add text cv2.putText(image, "THICC flower", (x + int(w/10),y + int(h/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255,255,255), 2) # Display cv2.imshow('image', image) cv2.waitKey()