eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I haven't found a way to only get exponents which are multiples of 3, when displaying numbers in the scientific format. Neither did I succeed writing a simple custom formatting function. Here is a quick example: Normal behaviour using scientific notation with pythons .format(): numbers = [1.2e-2, 1.3e-3, 1.5e5, 1.6e6] for n in numbers: print("{:.E}".format(n))  >>> 1.20E-02 1.30E-03 1.50E+05 1.60E+06  I, however, need the following output: >>> 12.00E-03 1.30E-03 15.00E+06 1.60E+06  Does anyone know a convenient way for me to get the desired formatting?

Answer: Well, it depends on if you want the output format to always adjust to the nearest power of 3, or if you want it to adjust to the nearest lower power of 3. Basically it comes to: how you handle 1.50E+05? Should it be 150.00E+03 or 0.15E+06? Case 1: nearest lower power of 3 from math import log10,floor numbers = [1.2e-2, 1.3e-3, 1.5e5, 1.6e6] def adjusted_scientific_notation(val,num_decimals=2,exponent_pad=2): exponent_template = "{:0>%d}" % exponent_pad mantissa_template = "{:.%df}" % num_decimals order_of_magnitude = floor(log10(abs(val))) nearest_lower_third = 3*(order_of_magnitude//3) adjusted_mantissa = val*10**(-nearest_lower_third) adjusted_mantissa_string = mantissa_template.format(adjusted_mantissa) adjusted_exponent_string = "+-"[nearest_lower_third<0] + exponent_template.format(abs(nearest_lower_third)) return adjusted_mantissa_string+"E"+adjusted_exponent_string for n in numbers: print("{0:.2E} -> {1: >10}".format(n,adjusted_scientific_notation(n)))  which prints out: 1.20E-02 -> 12.00E-03 1.30E-03 -> 1.30E-03 1.50E+05 -> 150.00E+03 1.60E+06 -> 1.60E+06  Case 2: nearest power of 3 def adjusted_scientific_notation(val,num_decimals=2,exponent_pad=2): exponent_template = "{:0>%d}" % exponent_pad mantissa_template = "{:.%df}" % num_decimals order_of_magnitude = floor(log10(abs(val))) nearest_third = 3*(order_of_magnitude//3+int(order_of_magnitude%3==2)) adjusted_mantissa = val*10**(-nearest_third) adjusted_mantissa_string = mantissa_template.format(adjusted_mantissa) adjusted_exponent_string = "+-"[nearest_third<0] + exponent_template.format(abs(nearest_third)) return adjusted_mantissa_string+"E"+adjusted_exponent_string for n in numbers: print("{0:.2E} -> {1: >10}".format(n,adjusted_scientific_notation(n)))  which prints out: 1.20E-02 -> 12.00E-03 1.30E-03 -> 1.30E-03 1.50E+05 -> 0.15E+06 1.60E+06 -> 1.60E+06
Question: I have following dataset Item Count A 60 A 20 A 21 B 33 B 33 B 32  Code to reproduce: import pandas as pd df = pd.DataFrame([ ['A', 60], ['A', 20], ['A', 21], ['B', 33], ['B', 33], ['B', 32], ], columns=['Item', 'Count'])  Suppose I have to Change only the maximum value of each group of "Item" column by adding 1. the output should be like this: Item Count New_Count A 60 61 A 20 20 A 21 21 B 33 34 B 33 34 B 32 32  I tried df['New_Count']=df.groupby(['Item'])['Count'].transform(lambda x: max(x)+1) but all the values in "Count" was replaced by max value of each group +1. Item Count New_Count A 60 61 A 20 61 A 21 61 B 33 34 B 33 34 B 32 34 

Answer: Use idxmax: idx = df.groupby("Item")["Count"].idxmax() df["New_Count"] = df["Count"] df.loc[idx, "New_Count"] += 1  This will only increment the first occurrence of th maximum in each group. If you want to increment all the maximum values in the case of a tie, you can use transform instead. Just replace the first line above with: idx = df.groupby("Item")["Count"].transform(max) == df["Count"]
Question: How do I install python3 and python3-pip on an alpine based image (without using a python image)?  $ apk add --update python3.8 python3-pip ERROR: unsatisfiable constraints: python3-pip (missing): required by: world[python3-pip] python3.8 (missing): required by: world[python3.8] 

Answer: This is what I use in a Dockerfile for an alpine image: # Install python/pip ENV PYTHONUNBUFFERED=1 RUN apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python RUN python3 -m ensurepip RUN pip3 install --no-cache --upgrade pip setuptools
Question: I have the following code: import torch import numpy as np import pandas as pd from torch.utils.data import TensorDataset, DataLoader # Load dataset df = pd.read_csv(r'../iris.csv') # Extract features and target data = df.drop('target',axis=1).values labels = df['target'].values # Create tensor dataset iris = TensorDataset(torch.FloatTensor(data),torch.LongTensor(labels)) # Create random batches iris_loader = DataLoader(iris, batch_size=105, shuffle=True) next(iter(iris_loader))  What does next() and iter() do in the above code? I have went through PyTorch's documentation and still can't quite understand what is next() and iter() doing here. Can anyone help in explaining this? Many thanks in advance.

Answer: These are built-in functions of python, they are used for working with iterables. Basically iter() calls the __iter__() method on the iris_loader which returns an iterator. next() then calls the __next__() method on that iterator to get the first iteration. Running next() again will get the second item of the iterator, etc. This logic often happens 'behind the scenes', for example when running a for loop. It calls the __iter__() method on the iterable, and then calls __next__() on the returned iterator until it reaches the end of the iterator. It then raises a stopIteration and the loop stops. Please see the documentation for further details and some nuances: https://docs.python.org/3/library/functions.html#iter
Question: Both typing and collections.abc includes similar type such as Mapping, Sequence, etc. Based on the python documentation, it seems that collections.abc is preferred for type checking:  This module provides abstract base classes that can be used to test whether a class provides a particular interface; for example, whether it is hashable or whether it is a mapping. https://docs.python.org/3/library/collections.abc.html  but using typing also works and I'd rather not import Mapping from both typing and collections.abc. So is there any catch in using typing with isinstance()?

Answer: Many of the typing generic classes are just aliases to the abc ones. Just as an example from the docs, Hashable:  class typing.Hashable An alias to collections.abc.Hashable  Also, isinstance(abc.Hashable, typing.Hashable) isinstance(typing.Hashable, abc.Hashable)  are both True, making it clear they are equivalent in terms of the class hierarchy. In Python you can check an alias's origin using the origin field: >>> typing.Hashable.__origin__ <class 'collections.abc.Hashable'>  No reason I can see to import abc if you do not need it. The two packages provide close but different uses, so I would import only what you need. This is more of an opinion, but if your use-case is only checking against isinstance (i.e. no interesting argument annotation) both seem equivalent to me, though abc may be the more usual tool for this.
Question: I wrote a juypter notebook that has been converted to .py somehow. I would like it back in the original format. Does anyone know how to do that? There is a previous stack overflow question about this, but the solution doesn't work for me. Converting to (not from) ipython Notebook format

Answer: The question was edited so this isn't as direct of an answer as it once was. Nevertheless, if you accidentally changed the extension of your Python notebook from .ipynb to .py as the OP did when originally asking the question, this is the answer for you. Just rename it changing the extension e.g. for linux/macos mv <file>.py <file>.ipynb  or right-click rename for windows and type the full name with the extension (Since it seems that the contents are .ipynb contents already)
Question: Is it possible to install AWS CLI v2 through PIP on Windows? In the instructions the recommended way to install is via MSI, but I want to use PIP. What if I install CLI like given on Github in a Linux way: python -m pip install awscli  Will it install v1 or v2 by default?

Answer: pip install awscliv2 This single command should help you install AWS CLI v2
Question: I am building a web form using Flask and would like the user to be able to enter multiple entries, and give them the opportunity to regret an entry with an undo button, before sending the data to the database. I am trying to use Flask-Caching but have not managed to set it up properly. I have followed The Flask Mega-Tutorial for setting up Flask (this is my first Flask app). +---app | | forms.py | | routes.py | | __init__.py | +---static | +---templates  I wonder how I need to configure the Flask app to basically be able to do the following things: cache.add("variable_name", variable_data) variable_name = cache.get("variable_name") cache.clear()  in one of the pages (functions with @app.route decorators)? In app.init.py I have: from flask import Flask from config import Config from flask_caching import Cache app = Flask(__name__) app.config.from_object(Config) cache = Cache(app, config={'CACHE_TYPE': 'simple'}) from app import routes  In routes.py I have: from flask import current_app  and I use code below when I try to call the cache. current_app.cache.add("variable_name", variable_data)  What I get when trying to use the form is the following error: AttributeError: 'Flask' object has no attribute 'cache'  Pretty much all tutorials I've found have simply had the app declaration and all the routes in the same module. But how do I access the cache when I have the routes in another module?

Answer: You first statement makes me wonder if you are really looking for caching. It seems you are looking for session data storage. Some possibilities... 1. Session Data Storage  Client-Side: Store session client data as cookies using built-in Flask session objects. From docs: This can be any small, basic information about that client or their interactions for quick retrieval (up to 4kB).  Example session storing explicitly variable: username from flask import Flask, session, request app = Flask(__name__) app.config["SECRET_KEY"] = "any random string" @app.route("/login", methods=["GET", "POST"]) def login(): if request.method == "POST": session["username"] = request.form["username"] # to get value use session["username"]   Server-Side: For configurations or long-standing session data that usually still don't require a database. A more comprehensive example usage look here for the Flask-Session package.  2. Caching (Flask-Caching package) Must be stated that:  A cache's primary purpose is to increase data retrieval performance by reducing the need to access the underlying slower storage layer  So if you need to spee-up things on your site... this is the recommended approach by the Flask team. Example of caching a variable username that will expire after CACHE_DEFAULT_TIMEOUT from flask import Flask, request from flask_caching import Cache app = Flask(__name__) app.config["SECRET_KEY"] = "any random string" app.config["CACHE_TYPE"] = "SimpleCache" app.config["CACHE_DEFAULT_TIMEOUT"] = 300 # timeout in seconds cache = Cache(app) @app.route("/login", methods=["GET", "POST"]) def login(): if request.method == "POST": cache.set("username", request.form["username"]) # to get value use cache.get("username")
Question: We need such a code for hashing: from hashlib import sha256 Hash = sha256(b"hello").hexdigest() #Hash = '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'  hexdigest seems to be doing the main thing, because without it we will get the following result: Hash = sha256(b"hello") #Hash = <sha256 HASH object @ 0x000001E92939B950>  The use of hexdigest is mandatory because if it is not used, another output will be obtained, but what does it do?

Answer: The actual digest is a really big number. It is conventionally represented as a sequence of hex digits, as we humans aren't very good at dealing with numbers with more than a handful of digits (and hex has the advantage that it reveals some types of binary patterns really well; for example, you'd be hard pressed to reason about a number like 4,262,789,120 whereas its hex representation FE150000 readily reveals that the low 16 bits are all zeros) but the object is more than just a number; it's a class instance with methods which allow you e.g. to add more data in chunks, so that you can calculate the digest of a large file or a stream of data successively, without keeping all of it in memory. You can think of the digest object as a collection of states which permit this operation to be repeated many times, and the hex digest method as a way to query its state at the current point in the input stream. You could argue that the interface could be different - for example, str(Hash) could produce the hex representation; but this only pushes the problem to a different, and arguably more obscure, corner. For completeness, hexdigest is not well-defined in Python generally. It is the name of a set of methods within the hashlib module in the standard library, and this exposition discusses that particular use case. Other libraries could have a method with the same name but completely different semantics; or they could have a method with the same purpose but a completely different name.
Question: I wonder how to reload Python package after pip install in Visual Studio Code?  pip install package-A pip list package-A does not exist Restart 'Visual Studio Code'  Is the only way to restart?

Answer: The best answer I've found is to use Developer: Reload Window in the command palette like @rioV8 suggested. You can either use the command palette or you can change the key mappings as described here. There's already a Ctrl + R key mapping for reloading the window, but it's got a 'when' condition attached to it so I changed that to true. I also had to delete key mappings for other extensions so there was no overlap.
Question: I am trying to generate blob SAS URLs on the fly using the azure-storage-blob package. This solution only works if you have the now-deprecated azure-storage package, which cannot be installed anymore. I need a way to mimic the behaviour of BlockBlobService.generate_blob_shared_access_signature to generate a blob SAS URL, like this: from datetime import datetime, timedelta from azure.storage.blob import ( BlockBlobService, ContainerPermissions, BlobPermissions, PublicAccess, ) AZURE_ACC_NAME = '<account_name>' AZURE_PRIMARY_KEY = '<account_key>' AZURE_CONTAINER = '<container_name>' AZURE_BLOB='<blob_name>' block_blob_service = BlockBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY) sas_url = block_blob_service.generate_blob_shared_access_signature(AZURE_CONTAINER,AZURE_BLOB,permission=BlobPermissions.READ,expiry= datetime.utcnow() + timedelta(hours=1)) print('https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_BLOB+'?'+sas_url)  The above solution works if you have the deprecated package, but I need a solution which doesn't need it.

Answer: Take a look to the following code: from datetime import datetime, timedelta from azure.storage.blob import BlobClient, generate_blob_sas, BlobSasPermissions account_name = 'STORAGE_ACCOUNT_NAME' account_key = 'STORAGE_ACCOUNT_ACCESS_KEY' container_name = 'CONTAINER_NAME' blob_name = 'IMAGE_PATH/IMAGE_NAME' def get_blob_sas(account_name,account_key, container_name, blob_name): sas_blob = generate_blob_sas(account_name=account_name, container_name=container_name, blob_name=blob_name, account_key=account_key, permission=BlobSasPermissions(read=True), expiry=datetime.utcnow() + timedelta(hours=1)) return sas_blob blob = get_blob_sas(account_name,account_key, container_name, blob_name) url = 'https://'+account_name+'.blob.core.windows.net/'+container_name+'/'+blob_name+'?'+blob  Check this documentation for more detail: link
Question: I am learning about multiprocessing in python. I have the following code snippet: import time import concurrent.futures def wait(seconds): print(f'Waiting {seconds} seconds...') time.sleep(seconds) return f'Done' if __name__ == "__main__": with concurrent.futures.ProcessPoolExecutor() as executor: p = executor.submit(wait,1) print(p.result())  Running it gives me this error: BrokenProcessPool Traceback (most recent call last) <ipython-input-19-8eff57e3a077> in <module> 9 with concurrent.futures.ProcessPoolExecutor() as executor: 10 p = executor.submit(do_something,1) ---> 11 print(p.result()) ~\.conda\envs\w\lib\concurrent\futures\_base.py in result(self, timeout) 433 raise CancelledError() 434 elif self._state == FINISHED: --> 435 return self.__get_result() 436 else: 437 raise TimeoutError() ~\.conda\envs\w\lib\concurrent\futures\_base.py in __get_result(self) 382 def __get_result(self): 383 if self._exception: --> 384 raise self._exception 385 else: 386 return self._result BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.  I am on a Windows computer and I have used if __name__ == "__main__": in my code. But I still get this error.

Answer: I got it to work! I saved the wait function in a separate python file called wait.py and imported it in jupyter notebook. wait.py: import time def wait(seconds): print(f'Waiting {seconds} seconds...') time.sleep(seconds) return f'Done'  ipynb file: import concurrent.futures import wait #import the wait file if __name__ == "__main__": with concurrent.futures.ProcessPoolExecutor() as executor: p = executor.submit(wait.wait,1) print(p.result())
Question: I've looked up several SO and other web pages but I haven't found anything that works. The script I wrote, opens a docx, changes some words and then saves it in a certain folder as a docx. However, I want it to save it as a pdf but I don't know how to. This is an example of the code I'm working with: # Opening the original document doc = Document('./myDocument.docx') # Some code which changes the doc # Saving the changed doc as a docx doc.save('/my/folder/myChangedDocument.docx')  The things I tried to do for it to save as a pdf: from docx2pdf import convert # This after it was saved as a docx convert('/my/folder/myChangedDocument.docx', '/my/folder/myChangedDocument.pdf')  But it says that Word needs permission to open the saved file and I have to select the file to give it the permission. After that, it just says: 0%| | 0/1 [00:03<?, ?it/s] {'input': '/my/folder/contractsomeVariable.docx', 'output': '/my/folder/contractsomeVariable.pdf', 'result': 'error', 'error': 'Error: An error has occurred.'}  And I tried to simply put .pdf instead of .docx after the document name when I saved it but that didn't work either as the module docx can't do that. So does someone know how I can save a docx as a pdf using Python?

Answer: you can use docx2pdf by making the changes first and then coverting. Use pip to install on mac (I am guessing you already have it but it is still good to include). pip install docx2pdf  Once docx2pdf is installed, you can your docx file in inputfile and put an empty .pdf file in outputfile. from docx2pdf import convert inputFile = "document.docx" outputFile = "document2.pdf" file = open(outputFile, "w") file.close() convert(inputFile, outputFile)
Question: I'm reading 'Fluent Python' by 'Luciano Ramalho' over and over, but I couldn't understand asyncio.sleep's behavior inside asyncio. Book says at one part:  Never use time.sleep in asyncio coroutines unless you want to block the main thread, therefore freezing the event loop and probably the whole application as well. (...) it should yield from asyncio.sleep(DELAY).  On the other part:  Every Blocking I/O function in the Python standard library releases the GIL (...) The time.sleep() function also releases the GIL.  As time.sleep() releases GIL codes on other thread can run, but blocks current thread. Since asyncio is single-threaded, I understand that time.sleep blocks asyncio loop. But, how asyncio.sleep() isn't blocking thread? Is it possible to not delay event loop and wait at the same time?

Answer: The function asyncio.sleep simply registers a future to be called in x seconds while time.sleep suspends the execution for x seconds. You can test how both behave with this small example and see how asyncio.sleep(1) doesn't actually give you any clue on how long it will "sleep" because it's not what it really does: import asyncio import time from datetime import datetime async def sleep_demo(): print("async sleep start 1s: ", datetime.now().time()) await asyncio.sleep(1) print("async sleep end: ", datetime.now().time()) async def I_block_everyone(): print("regular sleep start 3s: ", datetime.now().time()) time.sleep(3) print("regular sleep end: ", datetime.now().time()) asyncio.gather(*[sleep_demo(), I_block_everyone()])  This prints: async sleep start 1s: 04:46:55 regular sleep start 3s: 04:46:55 regular sleep end: 04:46:58 async sleep end: 04:46:58  The blocking call time.sleep prevent the event loop from scheduling the future that resumes sleep_demo. In the end, it gains control back only after approximately 3 seconds (even though we explicitly requested a 1 second async sleep). Now concerning "The time.sleep() function also releases the GIL.", this is not a contradiction as it will only allow another thread to execute (but the current thread will remain pending for x seconds). Somewhat both look a bit similar, in one case the GIL is released to make room for another thread, in asyncio.sleep, the event loop gains control back to schedule another task.
Question: Anyone knows keyboard shortcut to move cells up or down in Jupyter notebook? Cannot find the shortcut, any clues?

Answer: Further to honeybadger's response, you can see when you open up the Edit Command Mode shortcuts dialog box that there are no shortcuts defined for moving a cell up and down, by default:  I simply typed in my preferred combination Ctrl-Shift-Down and Ctrl-Shift-Up in the 'add shortcut' field, and pressed Enter. This is the same in Windows/Mac. Cheers!