eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe and 2 separate dictionaries. Both dictionaries have the same keys but have different values. dict_1 has key-value pairs where the values are unique ids that correspond with the dataframe df. I want to be able to use the 2 dictionaries and the unique ids from the dict_1 to append the values of dict_2 into the dataframe df. Example of dataframe df: col_1 col_2 id col_3 100 500 a1 478 785 400 a1 490 ... ... a1 ... ... ... a2 ... ... ... a2 ... ... ... a2 ... ... ... a3 ... ... ... a3 ... ... ... a3 ... ... ... a4 ... ... ... a4 ... ... ... a4 ...  Example of dict_1: 1:['a1', 'a3'], 2:['a2', 'a4'], 3:[...], 4:[...], 5:[...], .  Example of dict_2: 1:[0, 1], 2:[1, 1], 3:[...], 4:[...], 5:[...], .  I'm trying to append the data from dict_2 using id's from dict_1 into the main df. In a sense add the 2 values (or n values) from the lists of dict_2 as 2 columns (or n columns) into the df. Resultant df: col_1 col_2 id col_3 new_col_1 new_col_2 100 500 a1 478 0 1 785 400 a1 490 0 1 ... ... a1 ... 0 1 ... ... a2 ... 1 1 ... ... a2 ... 1 1 ... ... a2 ... 1 1 ... ... a3 ... 0 1 ... ... a3 ... 0 1 ... ... a3 ... 0 1 ... ... a4 ... 1 1 ... ... a4 ... 1 1 ... ... a4 ... 1 1 

Answer: IIUC, the keys in your two dictionaries are aligned. One way is to create a dataframe with a column id containing the values in dict_1 and 2 (in this case but can be more) columns from the values in dict_2 aligned on the same key. Then use merge on id to get the result back in df # the two dictionaries. note in dict_2 I added an element for the list in key 2 # to show it works for any number of columns dict_1 = {1:['a1', 'a3'],2:['a2', 'a4'],} dict_2 = {1:[0,1],2:[1,1,2]} #create a dataframe from dict_2, here it might be something easier but can't find it df_2 = pd.concat([pd.Series(vals, name=key) for key, vals in dict_2.items()], axis=1).T print(df_2) #index are the keys, and columns are the future new_col_x 0 1 2 1 0.0 1.0 NaN 2 1.0 1.0 2.0 #concat with the dict_1 once explode the values in the list, # here just a print to see what it's doing print (pd.concat([pd.Series(dict_1, name='id').explode(),df_2], axis=1)) id 0 1 2 1 a1 0.0 1.0 NaN 1 a3 0.0 1.0 NaN 2 a2 1.0 1.0 2.0 2 a4 1.0 1.0 2.0 # use previous concat, with a rename to change column names and merge to df df = df.merge(pd.concat([pd.Series(dict_1, name='id').explode(),df_2], axis=1) .rename(columns=lambda x: f'new_col_{x+1}' if isinstance(x, int) else x), on='id', how='left')  and you get print (df) col_1 col_2 id col_3 new_col_1 new_col_2 new_col_3 0 100 500 a1 478 0.0 1.0 NaN 1 785 400 a1 490 0.0 1.0 NaN 2 ... ... a1 ... 0.0 1.0 NaN 3 ... ... a2 ... 1.0 1.0 2.0 4 ... ... a2 ... 1.0 1.0 2.0 5 ... ... a2 ... 1.0 1.0 2.0 6 ... ... a3 ... 0.0 1.0 NaN 7 ... ... a3 ... 0.0 1.0 NaN 8 ... ... a3 ... 0.0 1.0 NaN 9 ... ... a4 ... 1.0 1.0 2.0 10 ... ... a4 ... 1.0 1.0 2.0 11 ... ... a4 ... 1.0 1.0 2.0
Question: What's the difference between the pass statement: def function(): pass  and 3 dots: def function(): ...  Which way is better and faster to execute(CPython)?

Answer: pass has been in the language for a very long time and is just a no-op. It is designed to explicitly do nothing. ... is a token having the singleton value Ellipsis, similar to how None is a singleton value. Putting ... as your method body has the same effect as for example: def foo(): 1  The ... can be interpreted as a sentinel value where it makes sense from an API-design standpoint, e.g. if you overwrite __getitem__ to do something special if Ellipsis are passed, and then giving foo[...] special meaning. It is not specifically meant as a replacement for no-op stubs, though I have seen it being used that way and it doesn't hurt either
Question: I tried Google Cloud Functions with Python and there was a problem with running it. It said: Error: could not handle the request I checked the logs, but there was no error, just a log message: Function execution took 16 ms, finished with status: 'crash'  When I simplified the function to a printout then it worked properly. Then I added raise Exception('test') before the printout to see if the exception is going to Stackdriver Errors, but it didn't, I got the finished with status: 'crash' message again only in the log. Is this normal behavior? Or is it a bug and instead of crash I should see the exception as an error in the log?

Answer: Quite rightly, as alluded to in the Comments, the crash seems buggy about Google Cloud Functions with Python. The issue was reported to the Internal Google Cloud Functions engineers and evaluation is still ongoing. You can monitor this link for fixes
Question: I'm trying to print to console before and after processing that takes a while in a Django management command, like this: import requests import xmltodict from django.core.management.base import BaseCommand def get_all_routes(): url = 'http://busopen.jeju.go.kr/OpenAPI/service/bis/Bus' r = requests.get(url) data = xmltodict.parse(r.content) return data['response']['body']['items']['item'] class Command(BaseCommand): help = 'Updates the database via Bus Info API' def handle(self, *args, **options): self.stdout.write('Saving routes ... ', ending='') for route in get_all_routes(): route_obj = Route( route_type=route['routeTp'], route_id=route['routeId'], route_number=route['routeNum']) route_obj.save() self.stdout.write('done.')  In the above code, Saving routes ... is expected to print before the loop begins, and done. right next to it when the loop completes so that it looks like Saving routes ... done. in the end. However, the former doesn't print until the loop completes, when both strings finally print at the same time, which is not what I expected. I found this question, where the answer suggests flushing the output i.e. self.stdout.flush(), so I added that to my code:  def handle(self, *args, **options): self.stdout.write('Saving routes ... ', ending='') self.stdout.flush() for route in get_all_routes(): route_obj = Route( route_type=route['routeTp'], route_id=route['routeId'], route_number=route['routeNum']) route_obj.save() self.stdout.write('done.')  Still, the result remains unchanged. What could have I done wrong?

Answer: The thing to keep in mind is you're using self.stdout (as suggested in the Django docs), which is BaseCommand's override of Python's standard sys.stdout. There are two main differences between the 2 relevant to your problem:  The default "ending" in BaseCommand's version of self.stdout.write() is a new-line, forcing you to use the ending='' parameter, unlike sys.stdout.write() that has an empty ending as the default. This in itself is not causing your problem. The BaseCommand version of flush() does not really do anything (who would have thought?). This is a known bug: https://code.djangoproject.com/ticket/29533  So you really have 2 options:  Not use BaseCommand's self.stdout but instead use sys.stdout, in which case the flush does work Force the stdout to be totally unbuffered while running the management command by passing the "-u" parameter to python. So instead of running python manage.py <subcommand>, run python -u manage.py <subcommand>  Hope this helps.
Question: I'm using strict type checks via pyright. When I have a method that returns a pytorch DataLoader, then pyright complains about my type definition:  Declared return type, "DataLoader[Unknown]", is partially unknown Pyright (reportUnknownVariableType)  Taking a look at the type stub from pytorch's DataLoader (reduced to the important parts): class DataLoader(Generic[T_co]): dataset: Dataset[T_co] @overload def __init__(self, dataset: Dataset[T_co], ...  As far as I can see, the generic type T_co of the DataLoader should be defined by the __init__ dataset parameter. Pyright also complains about my Dataset type definition:  Type of parameter "dataset" is partially unknown Parameter type is "Dataset[Unknown]" Pyright (reportUnknownParameterType)  Taking a look at the Dataset type stub: class Dataset(Generic[T_co]): def __getitem__(self, index: int) -> T_co: ...  shows to me that the type should be inferred by the return type of __getitem__. My dataset's type signature of __getitem__ looks like this: def __getitem__(self, index: int) -> Tuple[Tensor, Tensor]:  Based on this I would expect Dataset and DataLoader to be inferred as Dataset[Tuple[Tensor, Tensor]] and DataLoader[Tuple[Tensor, Tensor]] but that is not the case. My guess is that pyright fails to statically infer the types here. I thought I could define the type signature my self like this: Dataset[Tuple[Tensor, Tensor]]  but that actually results in my python script crashing with:  TypeError: 'type' object is not subscriptable  How can I properly define the type for Dataset and DataLoader?

Answer: Since there was no reply on this question I was not sure if it is actually a bug in pyright. I therefore opened this issue on the github repository: https://github.com/microsoft/pyright/issues/698 Eric Traut explained in detail what the issue is and that pyright is working as designed. I try to give the gist of the main points here. Problem explanation Pyright attempts to infer return types if they are not provided but if they are provided as in this case, they need to be fully typed. Pyright does not fill in missing parts of a given type annotation. For example, pyright will try to infer the return type for the following function definition: def get_dataset():  But if the return type is given as Dataset then that is the return type pyright expects. def get_dataset() -> Dataset:  In this case Dataset is a generic class that does not handle subscripting like Dataset[int]. In Python 3.7 (what we are using) the Python interpreter will evaluate these type annotations what leads to the mentioned exception. Solution As of Python 3.10 the Python interpreter will no longer evaluate type annotations and the following type annotation will just work: def get_dataset() -> Dataset[int]:  As of Python 3.7 it is possible to enable this behavior via the following import: from __future__ import annotations  This is documented in PEP 563. You will also need to disable the rule E1136 for pylint to not warn about "unsubscriptable-object". Another workaround is to quote the type definition like this: def get_dataset() -> "Dataset[int]":
Question: I am setting poetry to create virtual environments in the project directory. I entered:  poetry config settings.virtualenvs.in-project true  and received error  [ValueError] Setting settings.virtualenvs.in-project does not exist  Also there is the text   home/alex/.poetry/lib/poetry/_vendor/py2.7/subprocess32.py:149: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads. "program uses threads.", RuntimeWarning)   How can I solve the error? It seems that error deals with version of the python. I am using Ubuntu 16.04 version Poetry version 1.0.5

Answer: The config has changed with the release of poetry 1.0. The prefix settings is no longer needed. So just type poetry config virtualenvs.in-project true. Concerning the subprocess warning: This seems to be just a warning and has no influence on the correct working of poetry. Also have a look at my comment in poetry's issue tracker. @ptd: poetry can work with python2 and python3.
Question: I'm having trouble getting the VS Code PyTest code runner to work the way I'd like. It seems pytest options may be an all-or-nothing situation. Is there any way to run different sets of PyTest options easily in the VS Code interface? For example:  By default, run all tests not marked with @pytest.mark.slow.  This can be done with the argument -m "not slow" But, if I put that in a pytest.ini file, then it will never run any tests marked slow, even if I pick that particular test in the interface and try to run it. The resulting output is collected 1 item... 1 item deselected.  Run sometimes with coverage enabled, and sometimes without.  The only way I can see to do this is to run PyTest from the command line, which then loses the benefit of auto-discovery, running/debugging individual tests from the in-line interface, etc. What am I missing? Note: Currently using VS Code 1.45.1, Python 3.7.6, and PyTest 5.3.5

Answer: You're not missing anything. There currently isn't a way to provide per-execution arguments to get the integration you want with the Test Explorer.
Question: PROBLEM DESCRIPTION  I'm setting up a new MacBook and decided to jump too fast into downloading Python 3.8. I downloaded it from the website https://www.python.org/ before realizing it's better practice to do so with homebrew. GOAL - Remove Python 3.8 from my PATH to later install with Homebrew I cleared Python 3.8 from my filesystem thanks to this page https://nektony.com/how-to/uninstall-python-on-mac, but the path the Version 3.8 is still in my PATH variable. Typing echo $PATH in my terminal (zsh) returns /Library/Frameworks/Python.framework/Versions/3.8/bin along with other paths. Does anyone know how I can remove this path? It no longer exists in my filesystem so it's pointing to nothing.  WHAT I HAVE TRIED I have checked all the following files using nano and they all do not have the export command which would place it in the path in the first place. Files checked:  /etc/profile /etc/bashrc ~/.bash_profile ~/.bash_login ~/.profile ~/.bashrc  MY ENV I am running a macbook pro with Catalina (10.15.4) and using zsh as my terminal. Any help is appreciated, thanks a lot!!

Answer: Found the solution! Through running grep {subset of the path you're trying to remove} . (don't forget the period at the end), I found all places where that path was found on my computer. That brought me to seeing that the ./.zprofile file was exporting the Python 3.8 path. I removed it from that file, saved it and restarted my Terminal. Now, the path is gone and I am happy
Question: Unfortunately, Chromedriver always is version-specific to the Chrome version you have installed. So when you pack your python code AND a chromedriver via PyInstaller in a deployable .exe-file for Windows, it will not work in most cases as you won't be able to have all chromedriver versions in the .exe-file. Anyone knows a way on how to download the correct chromedriver from the website automatically? If not, I'll come up with a code to download the zip-file and unpack it to temp. Thanks!

Answer: Here is the other solution, where webdriver_manager does not support. This script will get the latest chrome driver version downloaded. import requests import wget import zipfile import os # get the latest chrome driver version number url = 'https://chromedriver.storage.googleapis.com/LATEST_RELEASE' response = requests.get(url) version_number = response.text # build the donwload url download_url = "https://chromedriver.storage.googleapis.com/" + version_number +"/chromedriver_win32.zip" # download the zip file using the url built above latest_driver_zip = wget.download(download_url,'chromedriver.zip') # extract the zip file with zipfile.ZipFile(latest_driver_zip, 'r') as zip_ref: zip_ref.extractall() # you can specify the destination folder path here # delete the zip file downloaded above os.remove(latest_driver_zip)
Question: I have a dataframe column with the following format:  col1 col2 A [{'Id':42,'prices':['30',’78’]},{'Id': 44,'prices':['20','47',‘89’]}] B [{'Id':47,'prices':['30',’78’]},{'Id':94,'prices':['20']},{'Id':84,'prices':['20','98']}]  How can I transform it to the following ?  col1 Id price A 42 ['30',’78’] A 44 ['20','47',‘89’] B 47 ['30',’78’] B 94 ['20'] B 84 ['20','98']  I was thinking of using apply and lambda as a solution but I am not sure how.  Edit : In order to recreate this dataframe I use the following code : data = [['A', "[{'Id':42,'prices':['30','78']},{'Id': 44,'prices':['20','47','89']}]"], ['B', "[{'Id':47,'prices':['30','78']},{'Id':94,'prices':['20']},{'Id':84,'prices':['20','98']}]"]] df = pd.DataFrame(data, columns = ['col1', 'col2']) 

Answer: Solution if there are lists in column col2: print (type(df['col2'].iat[0])) <class 'list'> L = [{**{'col1': a}, **x} for a, b in df[['col1','col2']].to_numpy() for x in b] df = pd.DataFrame(L) print (df) col1 Id prices 0 A 42 [30, 78] 1 A 44 [20, 47, 89] 2 B 47 [30, 78] 3 B 94 [20] 4 B 84 [20, 98]  If there are strings: print (type(df['col2'].iat[0])) <class 'str'> import ast L = [{**{'col1': a}, **x} for a, b in df[['col1','col2']].to_numpy() for x in ast.literal_eval(b)] df = pd.DataFrame(L) print (df) col1 Id prices 0 A 42 [30, 78] 1 A 44 [20, 47, 89] 2 B 47 [30, 78] 3 B 94 [20] 4 B 84 [20, 98]  For better understanding is possible use: import ast L = [] for a, b in df[['col1','col2']].to_numpy(): for x in ast.literal_eval(b): d = {'col1': a} out = {**d, **x} L.append(out) df = pd.DataFrame(L) print (df) col1 Id prices 0 A 42 [30, 78] 1 A 44 [20, 47, 89] 2 B 47 [30, 78] 3 B 94 [20] 4 B 84 [20, 98]
Question: What are the advantages and disadvantages of using Pydantic's dataclass vs BaseModel? Are there any performance issues or is it easier to Pydantic's dataclass in the other python module?

Answer: Your question is answered in Pydantic's documentation, specifically:  Keep in mind that pydantic.dataclasses.dataclass is a drop-in replacement for dataclasses.dataclass with validation, not a replacement for pydantic.BaseModel (with a small difference in how initialization hooks work). There are cases where subclassing pydantic.BaseModel is the better choice. For more information and discussion see samuelcolvin/pydantic#710.  The discussion link will give you some of the context you are looking for. In general, Pydantic's BaseModel implementation is not bound to behave the same as Python's dataclass implementation. The example cited in the issue above is one good example: from pydantic import BaseModel from pydantic.dataclasses import dataclass from typing import List @dataclass class A: x: List[int] = [] # Above definition with a default of `[]` will result in: # ValueError: mutable default <class 'list'> for field x is not allowed: use default_factory # If you resolve this, the output will read as in the comments below. class B(BaseModel): x: List[int] = [] print(A(x=[1, 2]), A(x=[3, 4])) # Output: A(x=[1, 2]) A(x=[3, 4]) print(B(x=[1, 2]), B(x=[3, 4])) # Output: x=[1, 2] x=[3, 4]  If what you want first and foremost is dataclass behavior and then to simply augment it with some Pydantic validation features, the pydantic.dataclasses.dataclass approach may be what you want. Otherwise, BaseModel is probably what you want.
Question: I am using an online jupyter notebook that is somehow configured to read all .py files as jupyter notebook files:  I am a big fan of this setup and would like to use it everywhere. On my own jupyter installation however, .py files are just interpreted as test files and are not by default loaded into jupyter cells. How can I achieve the same configuration for my jupyter notebook?

Answer: What you're looking for is jupytext. You just need to install it into python env from which you're running your jupyter notebooks: pip install jupytext --upgrade  And you get this:
Question: I'm training a CNN architecture to solve a regression problem using PyTorch where my output is a tensor of 20 values. I planned to use RMSE as my loss function for the model and tried to use PyTorch's nn.MSELoss() and took the square root for it using torch.sqrt() for that but got confused after obtaining the results.I'll try my best to explain why. It's obvious that for a batch-size bs my output tensor's dimensions would be [bs , 20].I tried to implement and RMSE function of my own :  def loss_function (predicted_x , target ): loss = torch.sum(torch.square(predicted_x - target) , axis= 1)/(predicted_x.size()[1]) #Taking the mean of all the squares by dividing it with the number of outputs i.e 20 in my case loss = torch.sqrt(loss) loss = torch.sum(loss)/predicted_x.size()[0] #averaging out by batch-size return loss  But the output of my loss_function() and how PyTorch implements it with nn.MSELoss() differed . I'm not sure whether my implementation is wrong or am I using nn.MSELoss() in the wrong way.

Answer: The MSE loss is the mean of the squares of the errors. You're taking the square-root after computing the MSE, so there is no way to compare your loss function's output to that of the PyTorch nn.MSELoss() function — they're computing different values. However, you could just use the nn.MSELoss() to create your own RMSE loss function as: loss_fn = nn.MSELoss() RMSE_loss = torch.sqrt(loss_fn(prediction, target)) RMSE_loss.backward()  Hope that helps.
Question: I have the following dataframe:  A B C A 1 3 0 B 3 2 5 C 0 5 4  All I want is shown below: my_list = [('A','A',1),('A','B',3),('A','C',0),('B','B',2),('B','C',5),('C','C',4)]  Thanks in advance!

Answer: IIUC, you can do: df.stack().reset_index().agg(tuple,1).tolist()   [('A', 'A', 1), ('A', 'B', 3), ('A', 'C', 0), ('B', 'A', 3), ('B', 'B', 2), ('B', 'C', 5), ('C', 'A', 0), ('C', 'B', 5), ('C', 'C', 4)]
Question: I have a POST FastAPI method. I do not want to construct a class nor query string. So, I decide to apply Body() method. @app.post("/test-single-int") async def test_single_int( t: int = Body(...) ): pass  This is the request POST http://localhost:8000/test-single-int/ { "t": 10 }  And this is the response HTTP/1.1 422 Unprocessable Entity date: Fri, 22 May 2020 10:00:16 GMT server: uvicorn content-length: 83 content-type: application/json connection: close { "detail": [ { "loc": [ "body", "s" ], "msg": "str type expected", "type": "type_error.str" } ] }  However, after trying with many samples, I found that they will not error if I have more than one Body(). For example, @app.post("/test-multi-mix") async def test_multi_param( s: str = Body(...), t: int = Body(...), ): pass  Request POST http://localhost:8000/test-multi-mix/ { "s": "test", "t": 10 }  Response HTTP/1.1 200 OK date: Fri, 22 May 2020 10:16:12 GMT server: uvicorn content-length: 4 content-type: application/json connection: close null  Does anyone have any idea about my implementation? Are there wrong? Is it not best practice? Or it is a bug?

Answer: It is not a bug, it is how Body behaves, it exists for "extending" request params how documentation outlines: class Item(BaseModel): name: str class User(BaseModel): username: str full_name: str = None @app.put("/items/{item_id}") async def update_item( *, item_id: int, item: Item, user: User, importance: int = Body(..., gt=0), q: str = None ): pass  Valid request body for this view would be: { "item": { "name": "Foo", "tax": 3.2 }, "user": { "username": "dave", "full_name": "Dave Grohl" }, "importance": 5 }  If you really want to use Body alone you must specify embed=True, this one works as expected: @app.put("/items/{item_id}") async def update_item( *, item_id:int, importance: int = Body(..., gt=0, embed=True), q: str = None ): pass