eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have multiple test files, each has an async fixture that looks like this:  @pytest.fixture(scope="module") def event_loop(request): loop = asyncio.get_event_loop_policy().new_event_loop() yield loop loop.close() @pytest.fixture(scope="module") async def some_fixture(): return await make_fixture()  I'm using xdist for parallelization. In addition I have this decorator: @toolz.curry def throttle(limit, f): semaphore = asyncio.Semaphore(limit) @functools.wraps(f) async def wrapped(*args, **kwargs): async with semaphore: return await f(*args, **kwargs) return wrapped  and I have a function uses it: @throttle(10) def f(): ...  Now f is being called from multiple test files, and I'm getting an exception telling me that I can't use the semaphore from different event loops. I tried moving to a session-level event loop fixture:  @pytest.fixture(scope="session", autouse=True) def event_loop(request): loop = asyncio.get_event_loop_policy().new_event_loop() yield loop loop.close()  But this only gave me:  ScopeMismatch: You tried to access the 'function' scoped fixture 'event_loop' with a 'module' scoped request object, involved factories  Is it even possible to have xdist + async fixture + semaphore working together?

Answer: Eventually got it to work using the following conftest.py: import asyncio import pytest @pytest.fixture(scope="session") def event_loop(): return asyncio.get_event_loop()
Question: I am following the TensorFlow 2 Object Detection API Tutorial on a Macbook Here's what I got when running the given script for converting xmls to TFrecords Traceback (most recent call last): File "generate_tfrecord.py", line 62, in <module> label_map_dict = label_map_util.get_label_map_dict(label_map) File "/usr/local/lib/python3.8/site-packages/object_detection/utils/label_map_util.py", line 164, in get_label_map_dict label_map = load_labelmap(label_map_path) File "/usr/local/lib/python3.8/site-packages/object_detection/utils/label_map_util.py", line 133, in load_labelmap label_map_string = fid.read() File "/usr/local/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 116, in read self._preread_check() File "/usr/local/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 78, in _preread_check self._read_buf = _pywrap_file_io.BufferedInputStream( TypeError: __init__(): incompatible constructor arguments. The following argument types are supported: 1. tensorflow.python._pywrap_file_io.BufferedInputStream(arg0: str, arg1: int) Invoked with: item { name: "cat" id: 1 } , 524288  My label map file contains the following item { id: 1 name: 'cat' } 

Answer: It seems the problem can be resolved by replacing label_map = label_map_util.load_labelmap(args.labels_path) label_map_dict = label_map_util.get_label_map_dict(label_map)  as label_map_dict = label_map_util.get_label_map_dict(args.labels_path)
Question: I have a pylint message (w0707) on this piece of code (from https://www.django-rest-framework.org/tutorial/3-class-based-views/): class SnippetDetail(APIView): """ Retrieve, update or delete a snippet instance. """ def get_object(self, pk): try: return Snippet.objects.get(pk=pk) except Snippet.DoesNotExist: raise Http404  the message is: Consider explicitly re-raising using the 'from' keyword  I don't quite understand how to act to correct the problem.

Answer: The link in the comment on your question above outlines the issue and provides a solution, but for clarity of those landing straight on this page like myself, without having to go off to another thread, read and gain context, here is the answer to your specific problem: TL;DR; This is simply solved by aliasing the Exception you are 'excepting' and refering to it in your second raise. Taking your code snippet above, see the bottom two lines, I've added 'under-carets' to denote what I've added. class SnippetDetail(APIView): """ Retrieve, update or delete a snippet instance. """ def get_object(self, pk): try: return Snippet.objects.get(pk=pk) except Snippet.DoesNotExist as snip_no_exist: # ^^^^^^^^^^^^^^^^ raise Http404 from snip_no_exist # ^^^^^^^^^^^^^^^^^^  Note: The alias can be any well formed string.
Question: I defined an optional cookie parameter and now want to check if the cookie was set. Unfortunately, the variable does not equal to None but to an empty Cookie object. How can I check the cookie object if it is set? Here's how I defined the cookie parameter: @app.route("/graphcall") def graphcall(request: Request, ads_id: Optional[str] = Cookie(None)): if ads_id: # Do stuff if the ads_id is set 

Answer: I assume you tried this via SwaggerUI. Setting Cookie values currently does not work via SwaggerUI due to browser security restrictions. @app.get("/items/") async def read_items(ads_id: Optional[str] = Cookie(None)): if ads_id: answer = "set to %s" % ads_id else: answer = "not set" return {"ads_id": answer}  works perfectly from command line with Fastapi 0.61.0 $ curl -X GET "http://127.0.0.1:8000/items/" -H "accept: application/json" -H "Cookie: ads_id=foobar" {"ads_id":"set to foobar"} $ curl -X GET "http://127.0.0.1:8000/items/" -H "accept: application/json" {"ads_id":"not set"}
Question: Imagine I have lines A, B, C, D, and E. I want lines A, B, and C to appear on the plotly line chart. I want the user to have the option to add lines D and E but D and E should be hidden by default. Any suggestions on how to do this? Example, how would I hide Australia by default. import plotly.express as px df = px.data.gapminder().query("continent=='Oceania'") fig = px.line(df, x="year", y="lifeExp", color='country') fig.show() 

Answer: You need to play with the parameter visible setting it as legendonly within every trace import plotly.express as px countries_to_hide = ["Australia"] df = px.data.gapminder().query("continent=='Oceania'") fig = px.line(df, x="year", y="lifeExp", color='country') fig.for_each_trace(lambda trace: trace.update(visible="legendonly") if trace.name in countries_to_hide else ()) fig.show()
Question: I need help with converting a nested list of dictionaries with a nested list of dictionaries inside of it to a dataframe. At the end, I want something that looks like (the dots are for other columns in between):  id | isbn | isbn13 | .... | average_rating| 30278752 |1594634025|9781594634024| .... |3.92 | 34006942 |1501173219|9781501173219| .... |4.33 |   review_stat =[{'books': [{'id': 30278752, 'isbn': '1594634025', 'isbn13': '9781594634024', 'ratings_count': 4832, 'reviews_count': 8435, 'text_reviews_count': 417, 'work_ratings_count': 2081902, 'work_reviews_count': 3313007, 'work_text_reviews_count': 109912, 'average_rating': '3.92'}]}, {'books': [{'id': 34006942, 'isbn': '1501173219', 'isbn13': '9781501173219', 'ratings_count': 4373, 'reviews_count': 10741, 'text_reviews_count': 565, 'work_ratings_count': 1005504, 'work_reviews_count': 2142280, 'work_text_reviews_count': 75053, 'average_rating': '4.33'}]}] 

Answer: If you key is always books pd.concat([pd.DataFrame(i['books']) for i in review_stat])  id isbn isbn13 ratings_count reviews_count text_reviews_count work_ratings_count work_reviews_count work_text_reviews_count average_rating 0 30278752 1594634025 9781594634024 4832 8435 417 2081902 3313007 109912 3.92 0 34006942 1501173219 9781501173219 4373 10741 565 1005504 2142280 75053 4.33  You can always reset the index if you need
Question: I'm following a PyTorch tutorial which uses the BERT NLP model (feature extractor) from the Huggingface Transformers library. There are two pieces of interrelated code for gradient updates that I don't understand. (1) torch.no_grad() The tutorial has a class where the forward() function creates a torch.no_grad() block around a call to the BERT feature extractor, like this: bert = BertModel.from_pretrained('bert-base-uncased') class BERTGRUSentiment(nn.Module): def __init__(self, bert): super().__init__() self.bert = bert def forward(self, text): with torch.no_grad(): embedded = self.bert(text)[0]  (2) param.requires_grad = False There is another portion in the same tutorial where the BERT parameters are frozen. for name, param in model.named_parameters(): if name.startswith('bert'): param.requires_grad = False  When would I need (1) and/or (2)?  If I want to train with a frozen BERT, would I need to enable both? If I want to train to let BERT be updated, would I need to disable both?  Additionaly, I ran all four combinations and found:  with torch.no_grad requires_grad = False Parameters Ran ------------------ --------------------- ---------- --- a. Yes Yes 3M Successfully b. Yes No 112M Successfully c. No Yes 3M Successfully d. No No 112M CUDA out of memory  Can someone please explain what's going on? Why am I getting CUDA out of memory for (d) but not (b)? Both have 112M learnable parameters.

Answer: This is an older discussion, which has changed slightly over the years (mainly due to the purpose of with torch.no_grad() as a pattern. An excellent answer that kind of answers your question as well can be found on Stackoverflow already. However, since the original question is vastly different, I'll refrain from marking as duplicate, especially due to the second part about the memory. An initial explanation of no_grad is given here:  with torch.no_grad() is a context manager and is used to prevent calculating gradients [...].  requires_grad on the other hand is used  to freeze part of your model and train the rest [...].  Source again the SO post. Essentially, with requires_grad you are just disabling parts of a network, whereas no_grad will not store any gradients at all, since you're likely using it for inference and not training. To analyze the behavior of your combinations of parameters, let us investigate what is happening:  a) and b) do not store any gradients at all, which means that you have vastly more memory available to you, no matter the number of parameters, since you're not retaining them for a potential backward pass. c) has to store the forward pass for later backpropagation, however, only a limited number of parameter (3 million) are stored, which makes this still manageable. d), however, needs to store the forward pass for all 112 million parameters, which causes you to run out of memory.
Question: How do you use argparse to provide a list of arguments from a group of choices. For example, lets say I want something like: python sample.py --p1 ['a','b'] --p2 ['x','y']  Where p1 can only be any or all from list of 'a', 'b', 'c' and p2 can only be any or all from list of 'x', 'y', 'z'

Answer: I found a way to get the behavior you want, but with a different syntax than what you present. You have to specify each choice with a unique parameter name/value pair. If that's ok, then the following works: parser = argparse.ArgumentParser(prog='game.py') parser.add_argument('--p1', choices=['a', 'b', 'c'], action='append') args = parser.parse_args(['--p1', 'a', '--p1', 'b']) print(args)  Result: Namespace(p1=['a', 'b'])  but this fails appropriately: parser = argparse.ArgumentParser(prog='game.py') parser.add_argument('--p1', choices=['a', 'b', 'c'], action='append') args = parser.parse_args(['--p1', 'a', '--p1', 'b', '--p1', 'x']) print(args)  Result: usage: game.py [-h] [--p1 {a,b,c}] game.py: error: argument --p1: invalid choice: 'x' (choose from 'a', 'b', 'c')  I can find nothing in the docs that suggests that ArgumentParser will take a list of valid values as a parameter value, which is what your version would require.
Question: I am currently trying to install libpq-dev to install psycopg2. The problem is, when I try to install it, an error occurs saying I don't have the latest libpq5 version. However when I try to download the newer version of libpq5 the system says that I already have the latest version. An example of the error. lhmendes@lhmendes-GA-78LMT-S2P:~$ sudo apt-get install libpq-dev Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: libpq-dev : Depends: libpq5 (= 12.4-0ubuntu0.20.04.1) but 12.4-1.pgdg20.04+1 is to be installed E: Unable to correct problems, you have held broken packages. lhmendes@lhmendes-GA-78LMT-S2P:~$ sudo apt-get install libpq5 Reading package lists... Done Building dependency tree Reading state information... Done libpq5 is already the newest version (12.4-1.pgdg20.04+1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. 

Answer: I would say you have installed the latest libpq (12.4-1) but libpq-dev needs older version (12.4-0) and this makes problem. You may try to install older libpq  apt-get install libpq==12.4-0ubuntu0.20.04.1  but if other program uses the latest version then older version can make problem with this program. pgdg20 in 12.4-1.pgdg20.04+1 means it is not module from standard ubuntu repo but from some other repo - probably postgresql repo - and maybe this repo has also the latest version libpq-dev. You would search 12.4-1.pgdg20.04+1 in Google and maybe you could find also libpg-dev with 12.4-1.pgdg20.04+1  I found libpq-dev 12.4-1.pgdg20.04+1 and you can download .deb file and install it. Or you can add this postgresql repo and install with apt-get. This method will also inform about updates and then you could install updates automatically.
Question: Import libraries and models, from __future__ import print_function import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Flatten from keras.layers import Conv2D, MaxPooling2D import keras.backend as k batch_size = 128 num_classes = 10 epochs = 12  Below the written code, #Loss and Optimizer optimizer = keras.optimizers.Adam() loss = keras.losses.categorical_crossentropy()  Below the type error, which I badly faced and i can't make the solution, --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-8-f3fea941b382> in <module>() 1 #Loss and Optimizer 2 optimizer = keras.optimizers.Adam() ----> 3 loss = keras.losses.categorical_crossentropy() /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs) 199 """Call target, and fall back on dispatchers if there is a TypeError.""" 200 try: --> 201 return target(*args, **kwargs) 202 except (TypeError, ValueError): 203 # Note: convert_to_eager_tensor currently raises a ValueError, not a TypeError: categorical_crossentropy() missing 2 required positional arguments: 'y_true' and 'y_pred'  Need help to solve this problem, please help me. Advanced thanks.

Answer: This is the correct implementation for getting a Categorical Crossentropy class object. loss = keras.losses.CategoricalCrossentropy()  keras.losses.categorical_crossentropy this is a function which requires 2 parameters.
Question: I have a dataframe like this: df = pd.DataFrame( [['101', 'a', 'in', '10'], ['101', 'a', 'out', '10'], ['102', 'b', 'in', '20'], ['103', 'c', 'in', '30'], ['103', 'c', 'out', '40']], columns=['col1', 'col2', 'col3', 'col4'] )  I want to group by col1 and find paired records that have the same value in col2 and col4, but one has 'in' in col3 one has 'out' in col3. The expected outcome is: df_out = pd.DataFrame( [['101', 'a', 'in', '10'], ['101', 'a', 'out', '10']], columns=['col1', 'col2', 'col3', 'col4'] )  Thank you for the help.

Answer: Let us try transform with nunique out = df[df.groupby(['col1','col2','col4'])['col3'].transform('nunique')==2] Out[187]: col1 col2 col3 col4 0 101 a in 10 1 101 a out 10
Question: How can I catch exceptions from a process that was executed using multiprocessing.Process()? Consider the following python script that executes a simple failFunction() (which immediately throws a runtime error) inside of a child process using mulitprocessing.Process() #!/usr/bin/env python3 import multiprocessing, time # this function will be executed in a child process asynchronously def failFunction(): raise RuntimeError('trust fall, catch me!') # execute the helloWorld() function in a child process in the background process = multiprocessing.Process( target = failFunction, ) process.start() # <this is where async stuff would happen> time.sleep(1) # try (and fail) to catch the exception try: process.join() except Exception as e: print( "This won't catch the exception" )  As you can see from the following execution, attempting to wrap the .join() does not actually catch the exception user@host:~$ python3 example.py Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "example4.py", line 6, in failFunction raise RuntimeError('trust fall, catch me!') RuntimeError: trust fall, catch me! user@host:~$  How can I update the above script to actually catch the exception from the function that was executed inside of a child process using multiprocessing.Process()?

Answer: This can be achieved by overloading the run() method in the multiprocessing.Proccess() class with a try..except statement and setting up a Pipe() to get and store any raised exceptions from the child process into an instance field for named exception: #!/usr/bin/env python3 import multiprocessing, traceback, time class Process(multiprocessing.Process): def __init__(self, *args, **kwargs): multiprocessing.Process.__init__(self, *args, **kwargs) self._pconn, self._cconn = multiprocessing.Pipe() self._exception = None def run(self): try: multiprocessing.Process.run(self) self._cconn.send(None) except Exception as e: tb = traceback.format_exc() self._cconn.send((e, tb)) #raise e # You can still rise this exception if you need to @property def exception(self): if self._pconn.poll(): self._exception = self._pconn.recv() return self._exception # this function will be executed in a child process asynchronously def failFunction(): raise RuntimeError('trust fall, catch me!') # execute the helloWorld() function in a child process in the background process = Process( target = failFunction, ) process.start() # <this is where async stuff would happen> time.sleep(1) # catch the child process' exception try: process.join() if process.exception: raise process.exception except Exception as e: print( "Exception caught!" )  Example execution: user@host:~$ python3 example.py Exception caught! user@host:~$  Solution taken from this answer:  https://stackoverflow.com/a/33599967/1174102
Question: I am writing a post-api using fastapi. The required request-format is: { "leadid":LD123, "parties":[ { "uid":123123, "cust_name":"JOhn Doe", }, ...]}  The fastapi code in python is: class Customer(BaseModel): UID: str CustName: str class PackageIn(BaseModel): lead_id: str parties: Set[Customer] # threshold: Optional[int] = 85 app = FastAPI() @app.post('/') async def nm_v2(package:PackageIn): return {"resp":"Response"}  When I visit the SwaggerUI to submit the response, the error is "422 Error: Unprocessable Entity". Also, the SwaggerUI doc states { "detail": [ { "loc": [ "body" ], "msg": "unhashable type: 'Customer'", "type": "type_error" } ] }  I do not know how to create this dict() structure for request payload without creating a separate pydantic based class called Customer. Pl tell me how to rectify the error.

Answer: Pytdantic BaseClass is not hashable. There is a discussion about this feature, i guess it will not be implemented. There is workaround in the discussion, for your case you can try this: from pydantic import BaseModel from typing import Set class MyBaseModel(BaseModel): def __hash__(self): # make hashable BaseModel subclass return hash((type(self),) + tuple(self.__dict__.values())) class Customer(MyBaseModel): # Use hashable sublclass for your model UID: str CustName: str class PackageIn(BaseModel): lead_id: str parties: Set[Customer] # threshold: Optional[int] = 85 data = { "lead_id": 'LD123', "parties": [ { "UID": 123123, "CustName": "JOhn Doe", }]} PackageIn.parse_obj(data) # This part fastapi will make on post request, just for test > <PackageIn lead_id='LD123' parties={<Customer UID='123123' CustName='JOhn Doe'>}>
Question: How to terminate loop.run_in_executor with ProcessPoolExecutor gracefully? Shortly after starting the program, SIGINT (ctrl + c) is sent. def blocking_task(): sleep(3) async def main(): exe = concurrent.futures.ProcessPoolExecutor(max_workers=4) loop = asyncio.get_event_loop() tasks = [loop.run_in_executor(exe, blocking_task) for i in range(3)] await asyncio.gather(*tasks) if __name__ == "__main__": try: asyncio.run(main()) except KeyboardInterrupt: print('ctrl + c')  With max_workers equal or lesser than the the number of tasks everything works. But if max_workers is greater, the output of the above code is as follows: Process ForkProcess-4: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.8/concurrent/futures/process.py", line 233, in _process_worker call_item = call_queue.get(block=True) File "/usr/lib/python3.8/multiprocessing/queues.py", line 97, in get res = self._recv_bytes() File "/usr/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) KeyboardInterrupt ctrl + c  I would like to catch the exception (KeyboardInterrupt) only once and ignore or mute the other exception(s) in the process pool, but how?  Update extra credit:  Can you explain (the reason for) the multi exception? Does adding a signal handler work on Windows? If not, is there a solution that works without a signal handler? 

Answer: You can use the initializer parameter of ProcessPoolExecutor to install a handler for SIGINT in each process. Update: On Unix, when the process is created, it becomes a member of the process group of its parent. If you are generating the SIGINT with Ctrl+C, then the signal is being sent to the entire process group. import asyncio import concurrent.futures import os import signal import sys from time import sleep def handler(signum, frame): print('SIGINT for PID=', os.getpid()) sys.exit(0) def init(): signal.signal(signal.SIGINT, handler) def blocking_task(): sleep(15) async def main(): exe = concurrent.futures.ProcessPoolExecutor(max_workers=5, initializer=init) loop = asyncio.get_event_loop() tasks = [loop.run_in_executor(exe, blocking_task) for i in range(2)] await asyncio.gather(*tasks) if __name__ == "__main__": try: asyncio.run(main()) except KeyboardInterrupt: print('ctrl + c')  Ctrl-C shortly after start: ^CSIGINT for PID= 59942 SIGINT for PID= 59943 SIGINT for PID= 59941 SIGINT for PID= 59945 SIGINT for PID= 59944 ctrl + c