eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: If I have this data frame: d = {'col1': [1, np.nan, np.nan], 'col2': [1, np.nan, 1]} df = pd.DataFrame(data=d) col1 col2 0 1.0 1.0 1 NaN NaN 2 NaN 1.0  and want to drop only rows that are empty to produce the following: d = {'col1': [1, np.nan], 'col2': [1, 1]} df = pd.DataFrame(data=d) col1 col2 0 1.0 1 1 NaN 1  What is the best way to do this?

Answer: Check the docs page df.dropna(how='all')
Question: Im new to python and have looked at various stack overflow posts. i feel like this should work but it doesnt. How do you import a class from another file in python? This folder structure src/example/ClassExample src/test/ClassExampleTest  I have this class class ClassExample: def __init__(self): pass def helloWorld(self): return "Hello World!"  I have this test class import unittest from example import ClassExample class ClassExampleTest(unittest.TestCase): def test_HelloWorld(self): hello = ClassExample() self.assertEqual("Hello World!", hello.helloWorld()) if __name__ == '__main__': unittest.main()  When the unit test runs the object is None:  AttributeError: 'NoneType' object has no attribute 'helloWorld'  What's wrong with this? How do you import a class in python?

Answer: If you're using Python 3, then imports are absolute by default. This means that import example will look for an absolute package named example, somewhere in the module search path. So instead, you probably want a relative import. This is useful when you want to import a module that is relative the module doing the importing. In this case: from ..example.ClassExample import ClassExample  I'm assuming that your folders are Python packages, meaning that they contain __init__.py files. So your directory structure would look like this. src |-- __init__.py |-- example | |-- __init__.py | |-- ClassExample.py |-- test | |-- __init__.py | |-- ClassExampleTest.py
Question: I'm trying to run this code: import pyautogui import time from PIL import _imaging from PIL import Image import pytesseract time.sleep(5) captura = pyautogui.screenshot() codigo = captura.crop((872, 292, 983, 337)) codigo.save(r'C:\autobot_wwe_supercard\imagenes\codigo.png') time.sleep(2) pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract' print(pytesseract.image_to_string(r'D:\codigo.png'))  And this error pops up: ImportError: cannot import name 'imaging' from 'PIL' (C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\PIL_init.py) I installed pillow in console with pip install pillow I installed pytesseract in console with pip install pytesseract

Answer: It appears as if a lot of PIL ImportErrors can simply be fixed by uninstalling and reinstalling Pillow again according to this source and your specific problem can be found here. Try these three commands: pip uninstall PIL pip uninstall Pillow pip install Pillow
Question: Consider the following snippet of python code: x = 1 class Foo: x = 2 def foo(): x = 3 class Foo: print(x) # prints 3 Foo.foo()  As expected, this prints 3. But, if we add a single line to the above snippet, the behavior changes: x = 1 class Foo: x = 2 def foo(): x = 3 class Foo: x += 10 print(x) # prints 11 Foo.foo()  And, if we switch the order of the two lines in the above example, the result changes yet again: x = 1 class Foo: x = 2 def foo(): x = 3 class Foo: print(x) # prints 1 x += 10 Foo.foo()  I'd like to understand why this occurs, and more generally, understand the scoping rules that cause this behavior. From the LEGB scoping rule, I would expect that both snippets print 3, 13, and 3, since there is an x defined in the enclosing function foo().

Answer: Class block scope is special. It is documented here:  A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution with an exception that unbound local variables are looked up in the global namespace. The namespace of the class definition becomes the attribute dictionary of the class. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods — this includes comprehensions and generator expressions since they are implemented using a function scope.  Basically, class blocks do not "participate" in creating/using enclosing scopes. So, it is actually the first example that isn't working as documented. I think this is an actual bug. EDIT: OK, so actually, here's some more relevant documentation from the data model. I think it all is actually consistent with the documentation:  The class body is executed (approximately) as exec(body, globals(), namespace). The key difference from a normal call to exec() is that lexical scoping allows the class body (including any methods) to reference names from the current and outer scopes when the class definition occurs inside a function.  So class blocks do participate in using enclosing scopes, but for free variables (as is normal anyway). In the first piece of documentation that I'm quoting, the part about "unbound local variables are looked up in the global namespace" applies to variables that would normally be marked local by the complier. So, consider this notorious error, for example: x = 1 def foo(): x += 1 print(x) foo()  This would throw an unbound local error, but an equivalent class definition would print 2: x = 1 class Foo: x += 1 print(x)  Basically, if there is an assignment statement anywhere in a class block, it is "local", but it will check in the global scope if there is an unbound local instead of throwing the UnboundLocal error. Hence, in your first example, it isn't a local variable, it is simply a free variable, and the resolution goes through the normal rules. In your next two examples, you use an assignment statement, marking x as "local", and thus, it will be looked up in the global namespace in case it is unbound in the local one.
Question: I have a dataframe which represents a population, with each column denoting a different quality/ characteristic of that person. How can I get a sample of that dataframe/ population, which is representative of the population as a whole across all characteristics. Suppose I have a dataframe which represents a workforce of 650 people as follows: import pandas as pd import numpy as np c = np.random.choice colours = ['blue', 'yellow', 'green', 'green... no, blue'] knights = ['Bedevere', 'Galahad', 'Arthur', 'Robin', 'Lancelot'] qualities = ['wise', 'brave', 'pure', 'not quite so brave'] df = pd.DataFrame({'name_id':c(range(3000), 650, replace=False), 'favourite_colour':c(colours, 650), 'favourite_knight':c(knights, 650), 'favourite_quality':c(qualities, 650)})  I can get a sample of the above that reflects the distribution of a single column as follows: # Find the distribution of a particular column using value_counts and normalize: knight_weight = df['favourite_knight'].value_counts(normalize=True) # Add this to my dataframe as a weights column: df['knight_weight'] = df['favourite_knight'].apply(lambda x: knight_weight[x]) # Then sample my dataframe using the weights column I just added as the 'weights' argument: df_sample = df.sample(140, weights=df['knight_weight'])  This will return a sample dataframe (df_sample) such that: df_sample['favourite_knight'].value_counts(normalize=True) is approximately equal to df['favourite_knight'].value_counts(normalize=True)  My question is this: How can I generate a sample dataframe (df_sample) such that the above i.e.: df_sample[column].value_counts(normalize=True) is approximately equal to df[column].value_counts(normalize=True)  is true for all columns (except 'name_id') instead of just one of them? population of 650 with a sample size of 140 is approximately the sizes I'm working with so performance isn't too much of an issue. I'll happily accept solutions that take a couple of minutes to run as this will still be considerably faster than producing the above sample manually. Thank you for any help.

Answer: You create a combined feature column, weight that one and draw with it as weights: df["combined"] = list(zip(df["favourite_colour"], df["favourite_knight"], df["favourite_quality"])) combined_weight = df['combined'].value_counts(normalize=True) df['combined_weight'] = df['combined'].apply(lambda x: combined_weight[x]) df_sample = df.sample(140, weights=df['combined_weight'])   This will need an additional step of dividing by the count of the specific weight so sum up to 1 - see Ehsan Fathi post.
Question: Good evening, I am using python 3.9 and try to run a new FastAPI service on Windows 10 Pro based on the documentation on internet https://www.uvicorn.org/ i executed the following statements pip install uvicorn pip install uvicorn[standard]  create the sample file app.py from fastapi import FastAPI app = FastAPI() @app.get("/") async def root(): return {"message": "Hello World"}  But when i run the code below : uvicorn main:app --reload uvicorn : The term 'uvicorn' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify t hat the path is correct and try again. At line:1 char:1 + uvicorn + ~~~~~~~ + CategoryInfo : ObjectNotFound: (uvicorn:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException  I also add the path of Python in the envroment settings I also re-install Python 3.9 and make the default path for installation to c:\ProgramFiles\Python39 this path is also include now in the system enviroment and user enviroment settings.  if i run pip install uvicorn again it shows the following statement: λ pip install uvicorn Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: uvicorn in c:\users\username\appdata\roaming\python\python39\site-packages (0.12.2) Requirement already satisfied: h11>=0.8 in c:\users\username\appdata\roaming\python\python39\site-packages (from uvicorn) (0.11.0) Requirement already satisfied: click==7.* in c:\users\username\appdata\roaming\python\python39\site-packages (from uvicorn) (7.1.2) WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available. You should consider upgrading via the 'c:\program files\python39\python.exe -m pip install --upgrade pip' command.  Many thanks Erik

Answer: Python installs it's scripts in the scripts folder at the following path: c:\users\username\appdata\roaming\python\python39\scripts Place that path in the system and user environment variable. This will solve the problem.
Question: I want something like this: from datetime import datetime, timezone import pytz def convert_datetime_by_timezone(timestamp_dt, to_timezone): if isinstance(to_timezone, str): return timestamp_dt.astimezone(pytz.timezone(to_timezone)) elif isinstance(to_timezone, pytz.tzinfo.??????): return timestamp_dt.astimezone(to_timezone) else: raise Exception("Invalid timezone: '%s'" % str(to_timezone))  But each time I create a pytz timezone a new type of object gets created: >>> type(pytz.timezone("UTC")) <class 'pytz.UTC'> >>> type(pytz.timezone("Europe/Budapest")) <class 'pytz.tzfile.Europe/Budapest'>  What is the correct way to check this?

Answer: isinstance(x, pytz.BaseTzInfo) works for both cases
Question: Environment:  interpreter: Python 3.9.0 OS: macOS Big Sur  This simple code is running fine with no errors; however, no image is produced and nothing is displayed, and I'm forced to manually stop the code and interrupt it to exit, otherwise it just seems to run forever? This exact same code used to work fine on my windows laptop. Any clue? The code: import cv2 CV_cat = cv2.imread('Cat.jpg') cv2.imshow('displaymywindows', CV_cat) cv2.waitKey(1500) 

Answer: With Ubuntu 18.04 and Anaconda environment I was getting the error with Pycharm as  process finished with exit code 139 (interrupted by signal 11 sigsegv)  Solved it by pip install opencv-python-headless in addition to pip install opencv-python
Question: I was looking at the Python Package Index (PyPi) and noticed 2 very similar packages: opencv-contrib-python and opencv-python and wondering what the difference was. I looked at them and they had the exact same description and version numbers.

Answer: As per PyPi documentation: There are four different packages (see options 1, 2, 3 and 4 below): Packages for standard desktop environments:  Option 1 - Main modules package: pip install opencv-python  Option 2 - Full package (contains both main modules and contrib/extra modules): pip install opencv-contrib-python (check contrib/extra modules listing from OpenCV documentation)  Packages for server (headless) environments:  Option 3 - Headless main modules package: pip install opencv-python-headless Option 4 - Headless full package (contains both main modules and contrib/extra modules): pip install opencv-contrib-python-headless  Do not install multiple different packages in the same environment
Question: I installed Airflow 1.10.12 using Anaconda in one of my environments. But when I tried to follow the Quick Start Guide (at https://airflow.apache.org/docs/stable/start.html) I got the following error after accessing http://localhost:8080/admin/ Traceback (most recent call last): File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/flask/app.py", line 1953, in full_dispatch_request return self.finalize_request(rv) File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/flask/app.py", line 1970, in finalize_request response = self.process_response(response) File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/flask/app.py", line 2269, in process_response self.session_interface.save_session(self, ctx.session, response) File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/flask/sessions.py", line 379, in save_session response.set_cookie( File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/werkzeug/wrappers/base_response.py", line 468, in set_cookie dump_cookie( File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/werkzeug/http.py", line 1217, in dump_cookie raise ValueError("SameSite must be 'Strict', 'Lax', or 'None'.") ValueError: SameSite must be 'Strict', 'Lax', or 'None'.  I tried to set the cookie_samesite variable at the default_airflow.cfg file to either None, Lax or Strict but the error persists. Some details of my airflow environment: I installed Apache airflow in my workspace using the conda install -c anaconda airflow After that I create an environment on Anaconda and installed the airflow package within this environment. Then I opened the terminal and run airflow initdb and airflow webserver -p 8080. Then I went to webpage and saw the error. Thanks in advance for any help.

Answer: Bump werkzeug version to the following: pip install 'werkzeug<1.0.0'  For Airflow >=2.0.0 change the config (airflow.cfg) [webserver] cookie_samesite to use Lax (https://github.com/apache/airflow/blob/2.0.1/UPDATING.md#the-default-value-for-webserver-cookie_samesite-has-been-changed-to-lax).
Question: In the Python docs about Context Vars a Context::run method is described to enable executing a callable inside a context so changes that the callable perform to the context are contained inside the copied Context. Though what if you need to execute a coroutine? What are you supposed to do in order to achieve the same behavior? In my case, what I wanted was something like this to handle a transactional context with possible nested transactions: my_ctxvar = ContextVar("my_ctxvar") async def coro(func, transaction): token = my_ctxvar.set(transaction) r = await func() my_ctxvar.reset(token) # no real need for this, but why not either return r async def foo(): ctx = copy_context() # simplification to one case here: let's use the current transaction if there is one if tx_owner := my_ctxvar not in ctx: tx = await create_transaction() else: tx = my_ctxvar.get() try: r = await ctx.run(coro) # not actually possible if tx_owner: await tx.commit() except Exception as e: if tx_owner: await tx.rollback() raise from e return r 

Answer: As I already pointed out here, context variables are natively supported by asyncio and are ready to be used without any extra configuration. It should be noted that:  Сoroutines executed by the current task by means of await share the same context New spawned tasks by create_task are executed in the copy of parent task context.  Therefore, in order to execute a coroutine in a copy of the current context, you can execute it as a task: await asyncio.create_task(coro())  Small example: import asyncio from contextvars import ContextVar var = ContextVar('var') async def foo(): await asyncio.sleep(1) print(f"var inside foo {var.get()}") var.set("ham") # change copy async def main(): var.set('spam') await asyncio.create_task(foo()) print(f"var after foo {var.get()}") asyncio.run(main())  var inside foo spam var after foo spam
Question: I have a lambda authorizer that is written in Python. I know that with the following access policy I can return 200/403 : { "principalId": "yyyyyyyy", "policyDocument": { "Version": "2012-10-17", "Statement": [ { "Action": "execute-api:Invoke", "Effect": "Deny", "Resource": "*" } ] }, "context": { "stringKey": "value", "numberKey": "1", "booleanKey": "true" }, "usageIdentifierKey": "{api-key}" }  I'm trying to return 401 error if the customer didn't send any token, therefore I'm raising an exception : raise Exception("Unauthorized")  The problem with this solution is that the AWS lambda fails and then the execution is marked as a failed execution and not as a successful execution of the lambda. Is there any way to return 401 without failing the lambda ? Also tried the following like in lambda integration but didn't work: return {"statusCode": 401, "body" : "Unauthorized"} 

Answer: It really is ugly, but that's the only way to truly signal a 401, which means "I can't find your Authorization header or cookie or nothing, you have to authenticate to do that". A 403 is an explicit 👎 saying "I know who you are, you're Forbidden from doing that". It's an odd, ternary response that API Gateway needs here: 👍/👎/🤷, and this "throw a very specific exception" is one way to do it. So you can't customize the response with the authorizer lambda; you can only give a response document that says yay/nay, or throw your hands up and signal "I can't find any authentication material here". To customize the shape of that response to a client, you would use Gateway Responses. With this, you can customize the shape of the json (or whatever content-type, really) of your 401/403 responses. Now, with respect to raise Exception("Unauthorized") polluting your metrics, making ambiguous real errors vs this expected error, I agree, it kinda stinks. My only recommendation would be to log something ERROR level that you set up a Metric Filter to watch out for, and use that as your true "something's gone wrong" metric.
Question: Code: from typing import AsyncIterable import asyncio async def agen() -> AsyncIterable[str]: print('agen start') yield '1' yield '2' async def agenmaker() -> AsyncIterable[str]: print('agenmaker start') return agen() async def amain(): print('amain') async for item in agen(): pass async for item in await agenmaker(): pass # Error: async for item in agenmaker(): pass def main(): asyncio.get_event_loop().run_until_complete(amain()) if __name__ == '__main__': main()  As you can see, it is type-annotated, and contains an easy-to-miss error. However, neither pylint nor mypy find that error. Aside from unit tests, what options are there for catching such errors?

Answer: MyPy is perfectly capable of finding this issue. The problem is that unannotated functions are not inspected. Annotate the offending function as -> None and it is correctly inspected and rejected. # annotated with return type async def amain() -> None: print('amain') async for item in agen(): pass async for item in await agenmaker(): pass # Error: async for item in agenmaker(): # error: "Coroutine[Any, Any, AsyncIterable[str]]" has no attribute "__aiter__" (not async iterable) pass  If you want to eliminate such issues slipping through, use the flags --disallow-untyped-defs or --check-untyped-defs.   MyPy: Function signatures and dynamic vs static typing A function without type annotations is considered to be dynamically typed by mypy: def greeting(name): return 'Hello ' + name  By default, mypy will not type check dynamically typed functions. This means that with a few exceptions, mypy will not report any errors with regular unannotated Python.
Question: Basically, I have 2 threads, receive and send. I want to be able to type a message, and whenever I get a new message it just gets 'printed above the line I am typing in'. first what I thought would work, and you can just paste this it will run: import multiprocessing import time from reprint import output import time import random import sys def receiveThread(queue): i = 0 while True: queue.put(i) i+=1 time.sleep(0.5) def sendThread(queue): while True: a = sys.stdin.read(1) if (a != ""): queue.put(a) if __name__ == "__main__": send_queue = multiprocessing.Queue() receive_queue = multiprocessing.Queue() send_thread = multiprocessing.Process(target=sendThread, args=[send_queue],) receive_thread = multiprocessing.Process(target=receiveThread, args=[receive_queue],) receive_thread.start() send_thread.start() with output(initial_len=2, interval=0) as output_lines: while True: output_lines[0] = "Received: {}".format(str(receive_queue.get())) output_lines[1] = "Last Sent: {}".format(str(send_queue.get()))  But what happens here is that i cannot send data. The input doesn't give me an EOF unlike when I put a = input(), but it overwrites whatever I put in that line, so how can I wait for the input in one thread while the other one works? EXPECTED BEHAVOIR: first line goes Received: 0, 1, 2, 3, 4... second line goes [my input until I press enter, then my input] ACTUAL BEHAVIOR if i don't check if input != "" first line as expected only that the input overwrites the first couple of letters until it resets to Received second line is always empty, maybe bc stdin only is filled for that one time i press enter and then always returns empty? ACTUAL BEHAVIOR if i check if input != "" first line stays: received = 0 second line is just like whatever i enter, if i press enter it goes into a new line where i then enter stuff

Answer: Don't use the same socket for communicating with... itself. That may be possible to do, I'm not sure, but it certainly isn't normal. Instead make a socket pair, one for the sending thread, and one for the receiving thread, e.g. this works for me: import socket; import multiprocessing; def receiveThread(sock): while True: msg = sock.recv(1024) print(msg.decode("utf-8")) def sendThread(sock): while True: # msg=input("client: ") # input() is broken on my system :( msg="foo" sock.send(bytes(msg,"utf8")) pair = socket.socketpair() recieve_thread_socket = pair[0] send_thread_socket = pair[1] send_thread = multiprocessing.Process(target=sendThread, args=[recieve_thread_socket]) receive_thread = multiprocessing.Process(target=receiveThread,args=[send_thread_socket]) send_thread.start() receive_thread.start()