eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I know this sounds stupid, but I'm reading a programming book and they talk about how print() can return nothing (None). They use this code to explain it. a = 10 b = 15 c = print('a =', a, 'b=', b) print(c)  I get it, c isn't any data type that print() can take and, y'know, print it. c just has an empty value because it's not a valid data type. But what data type is c? What data type is None? If c isn't a string, integer, float, nor a boolean, what is it? Shouldn't None be its own data type? P.S. If I go to python and assign a variable None and print it, it recognises the data value and does not spit a name error. So theoretically, *None is its own data type, right? Oh, and why does Python not convert c to string and then print it?

Answer: None is (like literally everything else in Python besides keywords) an object. Meaning it is an instance of a class (or type if you will). None is an instance of NoneType, which you can find out, if you do this: print(type(None)) # <class 'NoneType'>  So yes, None has its own data type in Python. The class is special in the sense that it only ever has one instance: None. And yes, there is a string representation defined for the NoneType class. And the string representation is, well "None", which you see, when you do this: (all equivalent in this case) print(None) print(str(None)) print(repr(None))  PS: It is also worth stressing that None is in fact a singleton. Meaning there only ever is exactly that one instance of the NoneType class. This is why you can do identity comparisons with None, i.e.: if my_variable is None: ...  As opposed to equality comparisons (which of course work too) like this: if my_variable == None: ...  And when any function (including built-ins like print) have no explicit return statement or an empty one (like return without anything after it), they implicitly always return that one special None object.
Question: Is there a way to type annotate a function or variable in Python in such a way it allows both an enum or Literal formed form the attributes of the enum? from enum import Enum from typing import Literal class State(str, Enum): ENABLED = "enabled" DISABLED = "disabled" def is_enabled(state: State | Literal["enabled", "disabled"]) -> bool: if isinstance(state, str): state = State(state) return state == State.ENABLED  In other words, is there a way to obtain the alias for Literal["enabled", "disabled"] without having to rewrite all the keys of the enum?

Answer: I'm afraid there's no such way. The first thing that comes to mind is iterating over enum's values to build a Literal type won't work, because Literal cannot contain arbitrary expressions. So, you cannot specify it explicitly: # THIS DOES NOT WORK def is_enabled(state: State | Literal[State.ENABLED.value, State.DISABLED.value]) -> bool: ...  There's an open issue on GitHub with the related discussion. Basically, you could have hardcoded literals for every Enum, but in that case, you need to update it in accordance with Enum updates, and one day it will be messy. So, I would either stick with State | str annotation or just State and expect your function to accept only enums. Also, take into account that you do not need to explicitly create an Enum object to test its value, you can just write "enabled" == State.ENABLED as I mentioned in the comments.
Question: I was playing with scopes and namespaces and I found a weird behaviour which I'm not sure how to explain. Say we have a file called new_script.py with inside a = 0 def func(): import new_script #import itself new_script.a += 1 print(new_script.a) func() print(a)  when executing it prints 1 1 2 0  I didn't expect the last print of the number 0. From what I understand, it prints the first two 1 executing the self-import statement incrementing the global a, then it prints 2 because it increments again the global a from inside the function, but then why the last print is 0 instead of 2?

Answer: Well, this has lead down a very interesting rabit hole. So thanks you for that. Here are the key points:  Imports will not recurse. If it's imported once, it will execute the module level code, but it will not execute again if it's imported again. Hence you only see 4 values. Imports are singletons. If you try this code:  # singleton_test.py import singleton_test def func(): import singleton_test #import itself print(singleton_test.singleton_test == singleton_test) func()  It will print: True True   The imported singleton version of a module is different from the original ran version of the module  With this in mind, we can explore your code, by enriching it with a few more comments, particularly using __name__ which contains the name of the current module, which will be __main__ if the current module is what was ran originally: a = 0 print("start", __name__) def func(): print("Do import", __name__) import new_script #import itself new_script.a += 1 print(new_script.a, "func", __name__) func() print(a, "outr", __name__)  This will print start __main__ Do import __main__ start new_script Do import new_script 1 func new_script 1 outr new_script 2 func __main__ 0 outr __main__  This shows you quite well, given that the imported module is a singleton (but not the module that was ran), that you  first print 1 in the function after you incremented value inside the function inside the module then you print 1 at the end of the imported module then you print 2 after incrementing the value on the singleton on the original run code then finally you print 0 for the unchanged outer module that you originally ran, but have not touched.
Question: I am trying to convert multiple pdfs (10k +) to jpg images and extract text from them. I am currently using the pdf2image python library but it is rather slow, is there any faster/fastest library than this? from pdf2image import convert_from_bytes images = convert_from_bytes(open(path,"rb").read())  Note : I am using ubantu 18.04  CPU : 4 core 8 thread ( ryzen 3 3100)  memory : 8 GB

Answer: pyvips is a bit quicker than pdf2image. I made a tiny benchmark: #!/usr/bin/python3 import sys from pdf2image import convert_from_bytes images = convert_from_bytes(open(sys.argv[1], "rb").read()) for i in range(len(images)): images[i].save(f"page-{i}.jpg")  With this test document I see: $ /usr/bin/time -f %M:%e ./pdf.py nipguide.pdf 1991624:4.80  So 2GB of memory and 4.8s of elapsed time. You could write this in pyvips as: #!/usr/bin/python3 import sys import pyvips image = pyvips.Image.new_from_file(sys.argv[1]) for i in range(image.get('n-pages')): image = pyvips.Image.new_from_file(filename, page=i) image.write_to_file(f"page-{i}.jpg")  I see: $ /usr/bin/time -f %M:%e ./vpdf.py nipguide.pdf[dpi=200] 676436:2.57  670MB of memory and 2.6s elapsed time. They are both using poppler behind the scenes, but pyvips calls directly into the library rather than using processes and temp files, and can overlap load and save. You can configure pyvips to use pdfium rather than poppler, though it's a bit more work, since pdfium is still not packaged by many distributions. pdfium can be perhaps 3x faster than poppler for some PDFs. You can use multiprocessing to get a further speedup. This will work better with pyvips because of the lower memory use, and the fact that it's not using huge temp files. If I modify the pyvips code to only render a single page, I can use gnu parallel to render each page in a separate process: $ time parallel ../vpdf.py us-public-health-and-welfare-code.pdf[dpi=150] ::: {1..100} real 0m1.846s user 0m38.200s sys 0m6.371s  So 100 pages at 150dpi in 1.8s.
Question: I am pretty proficient in pycharm but it is the first time I stumble into this problem.  I created a conda environment Finding the conda executable which for me is in /home/my_username/.miniconda3/envs/py39/bin/python Adding it to pycharm results in:   I tried to search for this issue and error but the results didnt help. I am using fedora 36 if it is relevant. Edit: The output of which conda is: /home/my_username/.miniconda3/condabin/conda  Then trying to add it as the interpreter as suggested in Pycharm: Conda executable path is empty: 

Answer:  click the 'add interpreter'-'Add local interpreter'; click 'Conda Environment' on the left panel, browse and select 'yourAnacondaDir\Scripts\conda.exe'; click 'load Environment'; then two options: 'use existing environment' and 'create new environment' show up; click the first option; In the 'Use existing environment' box, select 'your_env'
Question: I am trying to test a lambda function locally, the function is created from the public docker image from aws, however I want to install my own python library from my github, according to the documentation AWS sam Build I have to add a variable to be taken in the Dockerfile like this:  Dockerfile  FROM public.ecr.aws/lambda/python:3.8 COPY lambda_preprocessor.py requirements.txt ./ RUN yum install -y git RUN python3.8 -m pip install -r requirements.txt -t . ARG GITHUB_TOKEN RUN python3.8 -m pip install git+https://${GITHUB_TOKEN}@github.com/repository/library.git -t .  And to pass the GITHUB_TOKEN I can create a .json file containing the variables for the docker environment.  .json file named env.json  { "LambdaPreprocessor": { "GITHUB_TOKEN": "TOKEN_VALUE" } }  And simply pass the file address in the sam build: sam build --use-container --container-env-var-file env.json Or directly the value without the .json with the command: sam build --use-container --container-env-var GLOBAL_ENV_VAR=TOKEN_VALUE My problem is that I don't get the GITHUB_TOKEN variable either with the .json file or by putting it directly in the command with --container-env-var GITHUB_TOKEN=TOKEN_VALUE Using sam build --use-container --container-env-var GLOBAL_ENV_VAR=TOKEN_VALUE --debug shows that it doesn't take it when creating the lambda image. The only way that has worked for me is to put the token directly in the Dockerfile not as an build argument.  Promt output  Building image for LambdaPreprocessor function Setting DockerBuildArgs: {} for LambdaPreprocessor function  Does anyone know why this is happening, am I doing something wrong? If you need to see the template.yaml this is the lambda definition.  template.yaml  LambdaPreprocessor: Type: AWS::Serverless::Function Properties: PackageType: Image Architectures: - x86_64 Timeout: 180 Metadata: Dockerfile: Dockerfile DockerContext: ./lambda_preprocessor DockerTag: python3.8-v1  I'm doing it with vscode and wsl 2 with ubuntu 20.04 lts on windows 10

Answer: I am having this issue too. What I have learned is that in the Metadata field there is DockerBuildArgs: that you can also add. Example:  Metadata: DockerBuildArgs: MY_VAR: <some variable>  When I add this it does make it to the DockerBuildArgs dict.
Question: Looking for general advice, as I'm not completely sure what is causing this behavior which I did not encounter until recently. I'm finding it quite annoying because it can delete imports if I comment out a line during development.

Answer: Adding the following to your settings.json file (you can access it on Windows with ctrl+shift+p followed by a search for settings) "editor.codeActionsOnSave": { ... [other settings] ... "source.organizeImports": false }  Note: ensure that "editor.codeActionsOnSave" is not defined elsewhere (aside from language specific places), as this prevented the fix from working the first time I tried it.
Question: How do I create a matrix of ascending integers that are arrayed like this example of N=6? 1 3 6 2 5 0 4 0 0  Here another example for N=13: 1 3 6 10 0 2 5 9 13 0 4 8 12 0 0 7 11 0 0 0 10 0 0 0 0  Also, the solution should perform well for large N values. My code import numpy as np N = 13 array_dimension = 5 x = 0 y = 1 z = np.zeros((array_dimension,array_dimension)) z[0][0] = 1 for i in range(2, N+1): z[y][x] = i if y == 0: y = (x + 1) x = 0 else: x += 1 y -= 1 print(z)  [[ 1. 3. 6. 10. 0.] [ 2. 5. 9. 0. 0.] [ 4. 8. 13. 0. 0.] [ 7. 12. 0. 0. 0.] [11. 0. 0. 0. 0.]]  works, but there must be a more efficient way. Most likely via Numpy, but I cannot find a solution.

Answer: The assignment can be completed in one step by simply transforming the index of the lower triangle: def fill_diagonal(n): assert n > 0 m = int((2 * n - 1.75) ** 0.5 + 0.5) '''n >= ((1 + (m - 1)) * (m - 1)) / 2 + 1 => 2n - 2 >= m ** 2 - m => 2n - 7 / 4 >= (m - 1 / 2) ** 2 => (2n - 7 / 4) ** (1 / 2) + 1 / 2 >= m for n > 0 => m = floor((2n - 7 / 4) ** (1 / 2) + 1 / 2) or n <= ((1 + m) * m) / 2 => (2n + 1 / 4) ** (1 / 2) - 1 / 2 <= m for n > 0 => m = ceil((2n + 1 / 4) ** (1 / 2) - 1 / 2) ''' i, j = np.tril_indices(m) i -= j ret = np.zeros((m, m), int) ret[i[:n], j[:n]] = np.arange(1, n + 1) return ret  Test: >>> for i in range(1, 16): ... print(repr(fill_diagonal(i)), end='\n\n') ... array([[1]]) array([[1, 0], [2, 0]]) array([[1, 3], [2, 0]]) array([[1, 3, 0], [2, 0, 0], [4, 0, 0]]) array([[1, 3, 0], [2, 5, 0], [4, 0, 0]]) array([[1, 3, 6], [2, 5, 0], [4, 0, 0]]) array([[1, 3, 6, 0], [2, 5, 0, 0], [4, 0, 0, 0], [7, 0, 0, 0]]) array([[1, 3, 6, 0], [2, 5, 0, 0], [4, 8, 0, 0], [7, 0, 0, 0]]) array([[1, 3, 6, 0], [2, 5, 9, 0], [4, 8, 0, 0], [7, 0, 0, 0]]) array([[ 1, 3, 6, 10], [ 2, 5, 9, 0], [ 4, 8, 0, 0], [ 7, 0, 0, 0]]) array([[ 1, 3, 6, 10, 0], [ 2, 5, 9, 0, 0], [ 4, 8, 0, 0, 0], [ 7, 0, 0, 0, 0], [11, 0, 0, 0, 0]]) array([[ 1, 3, 6, 10, 0], [ 2, 5, 9, 0, 0], [ 4, 8, 0, 0, 0], [ 7, 12, 0, 0, 0], [11, 0, 0, 0, 0]]) array([[ 1, 3, 6, 10, 0], [ 2, 5, 9, 0, 0], [ 4, 8, 13, 0, 0], [ 7, 12, 0, 0, 0], [11, 0, 0, 0, 0]]) array([[ 1, 3, 6, 10, 0], [ 2, 5, 9, 14, 0], [ 4, 8, 13, 0, 0], [ 7, 12, 0, 0, 0], [11, 0, 0, 0, 0]]) array([[ 1, 3, 6, 10, 15], [ 2, 5, 9, 14, 0], [ 4, 8, 13, 0, 0], [ 7, 12, 0, 0, 0], [11, 0, 0, 0, 0]])  For the case of large n, the performance is about 10 to 20 times that of the loop solution: %timeit fill_diagonal(10 ** 5) 1.63 ms ± 94.5 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) %timeit fill_diagonal_loop(10 ** 5) # OP's solution 25.1 ms ± 218 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Question: I am using VS code and have a venv folder for shared projects that lives outside of workspace/project folders. I want to change my workspace to use the interpreter within my AppData\local... folder (system installation of Python). I have been reading up on this but not found a solution to do this. How would I do this please?  I should note that the virtual environment is not active so I can not do this. In the terminal I see this showing the workspace folder & the virtual environment.  This is messing up my debugging and very frustrating. Edit - to try better explain when opening my interpreter list (see image below). Each workspace (red box) has the interpreter named under it (green box). I want to change the interpreter that is linked to the workspace.  Edit2 - solution - So I was able to fix this by doing something I had tried before but for some reason this time it worked. Which was selecting the 'Select at workspace level' option.  Then I selected the interpreter and this time it allowed me to browse for the interpreter on my computer which then updated the interpreter. Don't see what the problem was but it worked. 

Answer: You can use shortcuts "ctrl+shift+P" and type "Python: Clear Workspace Interpreter Settings" AND "Python: Select Interpreter" to change the environment.  By default, the Python extension looks for and uses the first Python interpreter it finds in the system path. To select a specific environment, use the Python: Select Interpreter command from the Command Palette (Ctrl+Shift+P).  You can refer to the docs for more details.
Question: I am having a problem when using tqdm.notebook progress bar in Jupyter (version 3.4.4). When I launch a for loop, instead of the progress bar, I get the following text as output: Input: from tqdm.notebook import tqdm for i in tqdm(range(100)): a = 1  Output: root: n: 0 total: 100 elapsed: 0.01399087905883789 ncols: null nrows: 29 prefix: "" ascii: false unit: "it" unit_scale: false rate: null bar_format: null postfix: null unit_divisor: 1000 initial: 0 colour: null  This started happening after I updated Jupyter to its latest version. The usual solutions regarding Node.js and ipywidgets (see this one) didn't do the job. tqdm is also in its last version (4.63.0).

Answer: I ran across this in a dockerized jupyterlab service. This fixed it for me: (Done in the Dockerfile):  pip install -U jupyterlab-widgets==1.1.1 pip install -U ipywidgets==7.7.2
Question:  I would like to extract logos from golf balls for further image processing. I have already tried different methods.  I wanted to use the grayscale value of the images to locate their location and then cut it out. Due to many different logos and a black border around the images, this method unfortunately failed. as my next approach I thought that I first remove the black background and then repeat the procedure from 1. but also without success because there is a dark shadow in the lower left corner and this is also recognized as the "logo" with the grayscale method. Covering the border further on the outside is not a solution, because otherwise logos that are on the border will also be cut away or only half of them will be detected. I used the edge detection algorithm Canny of the Open CV library. The detection looked very promising, but I was not able to extract only the logo from the detection, because the edge of the Golfball was also recognized.  Any solution is welcome. Please forgive my English. Also, I am quite a beginner in programming. Probably there is a very simple solution to my problem but I thank you in advance for your help. Here are 2 example images first the type of images from which the logos should be extracted and then how the image should look like after extraction. Thank you very much. Best regards T  

Answer: This is essentially "adaptive" thresholding, except this approach doesn't need to threshold. It adapts to the illumination, leaving you with a perfectly fine grayscale image (or color, if extended to do that).  median blur (large kernel size) to estimate ball/illumination division to normalize  illumination:  normalized (and scaled a bit):  thresholded with Otsu:  def process(im, r=80): med = cv.medianBlur(im, 2*r+1) with np.errstate(divide='ignore', invalid='ignore'): normalized = np.where(med <= 1, 1, im.astype(np.float32) / med.astype(np.float32)) return (normalized, med) normalized, med = process(ball1, 80) # imshow(med) # imshow(normalized * 0.8) ret, thresh = cv.threshold((normalized.clip(0,1) * 255).astype('u1'), 0, 255, cv.THRESH_BINARY + cv.THRESH_OTSU) # imshow(thresh)
Question: I have been looking for a proper data encryption library in python for a long while, today I needed it once again, cannot find anything, so is there any way to encrypt data using a user-set password, if I find something it's usually insecure, if I find a good solution it has no support for user-set passwords, meaning I'm stuck, any way to do it? Here's some pseudocode: import encryption encryptor: encryption.Crypt = encryption.Crypt("my secret password") encryptor.encrypt("hello this is my very secret string") # => 9oe gyu yp9q*(Y 28j encryptor.decrypt("9oe gyu yp9q*(Y 28j") # => hello this is my very secret string  I don't care if it's an object, for all I care it can also be a function which accepts the password: import encryption encryption.encrypt("hello this is my very secret string", "my secret password") # => 9oe gyu yp9q*(Y 28j encryption.decrypt("9oe gyu yp9q*(Y 28j", "my secret password") # => hello this is my very secret string  I don't mind the way it's encrypted or decrypted, I just want to have a way to do it :), I also don't care abt it's output, it can be binary, an object, a string, anything

Answer: Building on the answer from Sam Hartzog, below is an example which follows the logic described for PBES2 (Password Based Encryption Scheme 2) defined in RFC8018, Section 6.2. However, it stops short of encoding algorithm choices and parameters. #!/usr/bin/python import base64 import secrets from cryptography.fernet import Fernet from cryptography.hazmat.primitives import hashes from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC KDF_ALGORITHM = hashes.SHA256() KDF_LENGTH = 32 KDF_ITERATIONS = 120000 def encrypt(plaintext: str, password: str) -> (bytes, bytes): # Derive a symmetric key using the passsword and a fresh random salt. salt = secrets.token_bytes(16) kdf = PBKDF2HMAC( algorithm=KDF_ALGORITHM, length=KDF_LENGTH, salt=salt, iterations=KDF_ITERATIONS) key = kdf.derive(password.encode("utf-8")) # Encrypt the message. f = Fernet(base64.urlsafe_b64encode(key)) ciphertext = f.encrypt(plaintext.encode("utf-8")) return ciphertext, salt def decrypt(ciphertext: bytes, password: str, salt: bytes) -> str: # Derive the symmetric key using the password and provided salt. kdf = PBKDF2HMAC( algorithm=KDF_ALGORITHM, length=KDF_LENGTH, salt=salt, iterations=KDF_ITERATIONS) key = kdf.derive(password.encode("utf-8")) # Decrypt the message f = Fernet(base64.urlsafe_b64encode(key)) plaintext = f.decrypt(ciphertext) return plaintext.decode("utf-8") def main(): password = "aStrongPassword" message = "a secret message" encrypted, salt = encrypt(message, password) decrypted = decrypt(encrypted, password, salt) print(f"message: {message}") print(f"encrypted: {encrypted}") print(f"decrypted: {decrypted}")  Output: message: a secret message encrypted: b'gAAAAABjDlH2eaRZmB4rduBdNHUOITV5q4oelpnLRUgI_uyQyNpUyW8h3c2lZYS1MwMpRWIZposcZvag9si1pc4IEK83_CzyBdXF27Aop9WWS6ybxTg9BSo=' decrypted: a secret message
Question: I am reading chunks of data that is an API response using the following code: d = zlib.decompressobj(zlib.MAX_WBITS|16) # for gzip for i in range(0, len(data), 4096): chunk = data[i:i+4096] # print(chunk) str_chunk = d.decompress(chunk) str_chunk = str_chunk.decode() # print(str_chunk) if '"@odata.nextLink"' in str_chunk: ab = '{' + str_chunk[str_chunk.index('"@odata.nextLink"'):len(str_chunk)+1] ab = ast.literal_eval(ab) url = ab['@odata.nextLink'] return url  An example of this working is: "@odata.nextLink":"someurl?$count=true It works in most cases but sometimes this key value pair gets cut off and it appears something like this: "@odata.nextLink":"someurl?$coun I can play around with the number of bits in this line for i in range(0, len(data), 4096) but that doesn't ensure that in some cases the data doesn't cut off as the page sizes (data size) can be different for each page size. How can I ensure that this key value pair is never cut off. Also, note that this key value pair is the last line/ last key-value pair of the API response. P.S.: I can't play around with API request parameters. Even tried reading it backwards but this gives a header incorrect issue: for i in range(len(data), 0, -4096): chunk = data[i -4096: i] str_chunk = d.decompress(chunk) str_chunk = str_chunk.decode() if '"@odata.nextLink"' in str_chunk: ab = '{' + str_chunk[str_chunk.index('"@odata.nextLink"'):len(str_chunk)+1] ab = ast.literal_eval(ab) url = ab['@odata.nextLink'] #print(url) return url  The above produces the following error which is really strange: str_chunk = d.decompress(chunk) zlib.error: Error -3 while decompressing data: incorrect header check 

Answer: str_chunk is a contiguous sequence of bytes from the API response that can start anywhere in the response, and end anywhere in the response. Of course it will sometimes end in the middle of some semantic content. (New information from comment that OP neglected to put in question. In fact, still not in question. OP requires that entire uncompressed content not be saved in memory.) If "@odata.nextLink" is a reliable marker for what you're looking for, then keep the last two decompressed chunks, concatenate those, then look for that marker. Once found, continue to read more chunks, concatenating them, until you have the full content you're looking for.