eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe like df = pd.DataFrame({ 'level0': [0,1,2], 'level1': ['a', 'b', 'b'], 'level2':['x', 'x', 'x'], 'data': [0.12, 0.34, 0.45]} ).set_index(['level0', 'level1', 'level2'])      level0 level1 level 2 data     0 a x 0.12   1 b x 0.34   2 b x 0.56     If level0, level1, and level2 are the index levels, I want to access the data at (2, b) but keep the first two levels of labels. If I do df.loc[(2, 'b')] the output is     level2 data     x 0.56     but my desired output is     level0 level1 level 2 data     2 b x 0.56     How do I keep the levels 0 and 1 while using loc? I could add these levels back afterwards, but this is slightly annoying, and I'm doing this frequently enough to wonder if there's a one step solution.

Answer: You can use the output of MultiIndex.get_locs in iloc: >>> df.iloc[df.index.get_locs((2, 'b'))] data level0 level1 level2 2 b x 0.45
Question: I have some programm in which multiple processes try to finish some function. My aim now is to stop all the other processes after one process has successfully finished the function. The python program shown below unfortunately waits until all the processes successfully solved the question given in find function. How can I fix my problem? import multiprocessing import random FIND = 50 MAX_COUNT = 100000 INTERVAL = range(10) def find(process, initial, return_dict): succ = False while succ == False: start=initial while(start <= MAX_COUNT): if(FIND == start): return_dict[process] = f"Found: {process}, start: {initial}" succ = True break; i = random.choice(INTERVAL) start = start + i print(start) processes = [] manager = multiprocessing.Manager() return_code = manager.dict() for i in range(5): process = multiprocessing.Process(target=find, args=(f'computer_{i}', i, return_code)) processes.append(process) process.start() for process in processes: process.join() print(return_code.values())  output can be for example: ['Found: computer_0, start: 0', 'Found: computer_4, start: 4', 'Found: computer_2, start: 2', 'Found: computer_1, start: 1', 'Found: computer_3, start: 3']  But this output shows me the program is waiting until all processes are finished ...

Answer: Use an Event to govern if the processes should keep running. Basically, it replaces succ with something that works over all processes. import multiprocessing import random FIND = 50 MAX_COUNT = 1000 def find(process, initial, return_dict, run): while run.is_set(): start = initial while start <= MAX_COUNT: if FIND == start: return_dict[process] = f"Found: {process}, start: {initial}" run.clear() # Stop running. break start += random.randrange(0, 10) print(start) if __name__ == "__main__": processes = [] manager = multiprocessing.Manager() return_code = manager.dict() run = manager.Event() run.set() # We should keep running. for i in range(5): process = multiprocessing.Process( target=find, args=(f"computer_{i}", i, return_code, run) ) processes.append(process) process.start() for process in processes: process.join() print(return_code.values())  Note that using __name__ is mandatory for multiprocessing to work properly when the process-creation method is set to 'spawn' which is the default on ms-windows and macOS but also available on linux. On those systems, the main module is imported into newly created Python processes. This needs to happen without side effects such as starting a process, and the __name__ mechanism ensures that.
Question: I have a .csv file that I would like to render in a FastAPI app. I only managed to render the .csv file in JSON format as follows: def transform_question_format(csv_file_name): json_file_name = f"{csv_file_name[:-4]}.json" # transforms the csv file into json file pd.read_csv(csv_file_name ,sep=",").to_json(json_file_name) with open(json_file_name, "r") as f: json_data = json.load(f) return json_data @app.get("/questions") def load_questions(): question_json = transform_question_format(question_csv_filename) return question_json  When I tried returning directly pd.read_csv(csv_file_name ,sep=",").to_json(json_file_name), it works, as it returns a string. How should I proceed? I believe this is not the good way to do it.

Answer: The below shows four different ways of returning the data stored in a .csv file/Pandas DataFrame (for solutions without using Pandas DataFrame, have a look here). Related answers on how to efficiently return a large dataframe can be found here and here as well. Option 1 The first option is to convert the file data into JSON and then parse it into a dict. You can optionally change the orientation of the data using the orient parameter in the .to_json() method. Note: Better not to use this option. See Updates below. from fastapi import FastAPI import pandas as pd import json app = FastAPI() df = pd.read_csv("file.csv") def parse_csv(df): res = df.to_json(orient="records") parsed = json.loads(res) return parsed @app.get("/questions") def load_questions(): return parse_csv(df)   Update 1: Using .to_dict() method would be a better option, as it would return a dict directly, instead of converting the DataFrame into JSON (using df.to_json()) and then that JSON string into dict (using json.loads()), as described earlier. Example: @app.get("/questions") def load_questions(): return df.to_dict(orient="records")   Update 2: When using .to_dict() method and returning the dict, FastAPI, behind the scenes, automatically converts that return value into JSON using the Python standard json.dumps(), after converting it into JSON-compatible data first, using the jsonable_encoder, and then putting that JSON-compatible data inside of a JSONResponse (see this answer for more details). Thus, to avoid that extra processing, you could still use the .to_json() method, but this time, put the JSON string in a custom Response and return it directly, as shown below: from fastapi import Response @app.get("/questions") def load_questions(): return Response(df.to_json(orient="records"), media_type="application/json")    Option 2 Another option is to return the data in string format, using .to_string() method. @app.get("/questions") def load_questions(): return df.to_string()  Option 3 You could also return the data as an HTML table, using .to_html() method. from fastapi.responses import HTMLResponse @app.get("/questions") def load_questions(): return HTMLResponse(content=df.to_html(), status_code=200)  Option 4 Finally, you can always return the file as is using FastAPI's FileResponse. from fastapi.responses import FileResponse @app.get("/questions") def load_questions(): return FileResponse(path="file.csv", filename="file.csv")
Question: Reading up on pyproject.toml, python -m pip install, poetry, flit, etc - I have several questions regarding replacing setup.py with pyproject.toml. My biggest question was - how does a toml file replace a setup.py. Meaning, a toml file can't do everything a py file can. Reading into it, poetry and flit completely replace setup.py with pyproject.toml. While pip uses the pyproject.toml to specify the build tools, but then still uses the setup.py for everything else. A good example is, pip currently doesn't have a way to do entry points for console script directly in a toml file, but poetry and flit do.  https://flit.readthedocs.io/en/latest/pyproject_toml.html#scripts-section https://python-poetry.org/docs/pyproject/#scripts  My main question right now is; The point of pyproject.toml is to provide build system requirement. It is a metadata file. So wouldn't the ideal solution to be to use this file only to specify the build system requirements and still leverage the setup.py for everything else. I am confused because I feel like we're losing a lot to over come a fairly simple problem. By entirely doing way with the setup.py and replacing it with pyproject.toml, we lose a lot of helpful things we can do in a setup.py. We can't use a __version.py__, and we lose the ability to automatically create a universal wheel and sdist and upload our packages to PyPi using Twine. which we can currently do in the setup.py file. I'm just having a had time wrapping my head around why we would want to completely replace the setup.py with a metadata only file. It seems like using them together is the best of both worlds. We solve the chicken and the egg build system issue, and we get to retain a lot of useful things the setup.py can do. Wouldn't we need a setup.py to install in Dev mode anyway? Or maybe that is just a pip problem?

Answer: Currently I am investigating this feature too. I found this experimental feature explanation of setuptools which should just refer to the pyproject.toml without any need of setup.py in the end. Regarding dynamic behavior of setup.py, I figured out that you can set a dynamic behavior for fields under the [project] metadata dynamic = ["version"] [tool.setuptools.dynamic] version = {attr = "my_package.__version__"}  whereat the corresponding version in this example is set in, e.g. my_package.__init__.py __version__ = "0.1.0" __all__ = ["__version__"]  In the end, I guess that setuptools will cover the missing setup.py execution and places the necessary egg-links for the development mode.
Question: Hopefully the title isn't too misleading, I'm not sure the best way to phrase my question. I'm trying to create a (X, Y) coordinate data type in Python. Is there a way to create a "custom data type" so that I have an object with a value, but also some supporting attributes? So far I've made this simple class: class Point: def __init__(self, x, y): self.x = x self.y = y self.tuple = (x, y)  Ideally, I'd like to be able to do something like this: >>> p = Point(4, 5) >>> >>> my_x = p.x # can access the `x` attribute with "dot syntax" >>> >>> my_tuple = p # or can access the tuple value directly # without needing to do `.tuple`, as if the `tuple` # attribute is the "default" attribute for the object  NOTE I'm not trying to simply display the tuple, I know I can do that with the __repr__ method In a way, I'm trying to create a very simplified numpy.ndarray, because the ndarrays are a datatype that have their own attributes. I tried looking thru the numpy source to see how this is done, but it was way over my head, haha. Any tips would be appreciated!

Answer: I am not sure what you want to do with the tuple. p will always be an instance of Point. What you intend to do there won't work. If you just don't want to use the dot notation, you could use a namedtuple or a dataclass instead of a class. Then cast their instances to a tuple using tuple() and astuple().  Using a namedtuple and tuple(): from collections import namedtuple Point = namedtuple("Point", ["x", "y"]) p = Point(4, 5) x = p.x y = p.y xy = p # xy = tuple(p) not necessary since namedtuple is already a tuple  Note: namedtuple is immutable, i.e. you can't change x and y.  Using a dataclasses.dataclass and dataclasses.astuple(): from dataclasses import dataclass, astuple @dataclass class Point: x: int y: int p = Point(4, 5) x = p.x y = p.y xy = astuple(p)
Question: I am testing Plotly Dash as a possible dashboarding tool. I am trying to run one of the charts found in the documentation: https://plotly.com/python/bar-charts/ import dash from dash import dcc from dash import html from dash.dependencies import Input, Output import plotly.express as px df = px.data.tips() days = df.day.unique() app = dash.Dash(__name__) app.layout = html.Div([ dcc.Dropdown( id="dropdown", options=[{"label": x, "value": x} for x in days], value=days[0], clearable=False, ), dcc.Graph(id="bar-chart"), ]) @app.callback( Output("bar-chart", "figure"), [Input("dropdown", "value")]) def update_bar_chart(day): mask = df["day"] == day fig = px.bar(df[mask], x="sex", y="total_bill", color="smoker", barmode="group") return fig app.run_server(debug=True, port=8049)  When I run this I get an error. Here is the end of the trace callback: File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc zmq.error.ZMQError: Address already in use  As you can see from my example, I have already tried altering the port to avoid this error. I have tried many ports around 8050, but they all seem to be "already in use." My guess is that Dash reserves the port then tries to use it but sees that it's already reserved, not knowing that it was reserved for the process it was about to execute. Does anyone know how to fix this error?

Answer: If you are running it from jupyter-notebook or jupyter-lab, you should run the app server as: app.run_server(debug=True, port=8049, use_reloader=False)
Question: What is the equivalent of drop_duplicates() from pandas in polars? import polars as pl df = pl.DataFrame({"a":[1,1,2], "b":[2,2,3], "c":[1,2,3]}) df  Output: shape: (3, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 2 ┆ 1 │ │ 1 ┆ 2 ┆ 2 │ │ 2 ┆ 3 ┆ 3 │ └─────┴─────┴─────┘  Code: df.drop_duplicates(["a", "b"])  Delivers the following error: # AttributeError: 'DataFrame' object has no attribute 'drop_duplicates' 

Answer: The right function name is .unique() import polars as pl df = pl.DataFrame({"a":[1,1,2], "b":[2,2,3], "c":[1,2,3]}) df.unique(subset=["a","b"])  And this delivers the right output: shape: (2, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 2 ┆ 1 │ ├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤ │ 2 ┆ 3 ┆ 3 │ └─────┴─────┴─────┘
Question: So I'm trying to integrate plotly with my django app however I'm having an issue rendering a chart. I was using VSCode which did not pick up the dependency conflict. However when i started to use Pycharm. It said my Dash was version 1.11 which satisfies the django-plotly-dash but did not satisfy the dash_bootstrap_components which required 2.0.0 I have now installed Dash version 1.10 which conflicts with both apps just to show the error message below: Relevant error code ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following de pendency conflicts. django-plotly-dash 1.6.6 requires dash<1.21.0,>=1.11, but you have dash 1.10.0 which is incompatible. dash-bootstrap-components 1.0.3 requires dash>=2.0.0, but you have dash 1.10.0 which is incompatible.  Any help is appreciated Answer As django-plotly-dash is on the latest version, i've decided to install dash 1.20 and downgrade by dash-bootstrap-components to 0.13.0 (https://github.com/facultyai/dash-bootstrap-components/releases?page=2) This has worked like a charm.. weirdly - Pycharm has a reference error for the imports but visual studio code does not show any error and my program/script works perfectly. The pycharm import issue may be due to a setting in pycharm? idk

Answer: As django-plotly-dash is on the latest version, i've decided to install dash 1.20 and downgrade by dash-bootstrap-components to 0.13.0 (https://github.com/facultyai/dash-bootstrap-components/releases?page=2) This has worked like a charm.. weirdly - Pycharm has a reference error for the imports but visual studio code does not show any error and my program/script works perfectly. The pycharm import issue may be due to a setting in pycharm? idk
Question: I use EndeavourOS and have updated my system on February 17 2022 using sudo pacman -Syu  Eversince, when I run docker-compose, I get this error message:  [4221] Error loading Python lib '/tmp/_MEIgGJQGW/libpython3.7m.so.1.0': dlopen: libcrypt.so.1: cannot open shared object file: No such file or directory  Some forum threads suggested to reinstall docker-compose, which I did. I tried the following solution, but both without success: Python3.7: error while loading shared libraries: libpython3.7m.so.1.0 How can I resolve this issue?

Answer: The underlying issue here is that you use docker-compose instead of docker compose, which are two different binaries. docker-compose is also known as V1, and is deprecated since April 26, 2022. Since then, it does not receive updates or patches, other than high-severity security patches. So, to fix your issue, use docker compose instead of docker-compose. If you compare docker compose version and docker-compose version, you will see that this uses the newer docker compose and runs without an issue.
Question: I am trying to send two simple commands using subprocess.run & trying to store results in a variable then print it but for one arg the output is coming for subprocess.run & for other its empty Arg are "help" & "adb devices" command I am sending which returns the output result = subprocess.run("help", capture_output=True, text=True, universal_newlines=True) print(result.stdout)  but this command with a different arg is not returning result = subprocess.run("adb devices", capture_output=True, text=True, universal_newlines=True) print(result.stdout)  If I try the same command with subprocess.checkoutput it returns the output can anyone explain what exactly is going on here Is there any specific usage scenario's for these command's like when to use which one ? c = subprocess.check_output( "adb devices", shell=True, stderr=subprocess.STDOUT) print(c) output - b'List of devices attached\r\n\r\n' 

Answer: It is because from the python documentation here: run method run method accepts the first parameter as arguments and not string. So you can try passing the arguments in a list as: result = subprocess.run(['abd', 'devices'], capture_output=True, text=True, universal_newlines=True)  Also, check_output method accepts args but it has a parameter call "shell = True" Therefore, it works for multi-word args. If you want to use the run method without a list, add shell=True in the run method parameter. (I tried for "man ls" command and it worked).
Question: Minimal example of the class: from pydantic import BaseModel class AdaptedModel(BaseModel): def get_all_fields(self, alias=False): return list(self.schema(by_alias=alias).get("properties").keys()) class TestClass(AdaptedModel): test: str  The way it works: dm.TestClass.get_all_fields(dm.TestClass)  Is there a way to make it work without giving the class again? Desired way to get all field names: dm.TestClass.get_all_fields()  It would also work if the field names are assigned to an attribute. Just any way to make it make it more readable

Answer: What about just using __fields__: from pydantic import BaseModel class AdaptedModel(BaseModel): parent_attr: str class TestClass(AdaptedModel): child_attr: str TestClass.__fields__  Output: {'parent_attr': ModelField(name='parent_attr', type=str, required=True), 'child_attr': ModelField(name='child_attr', type=str, required=True)}  This is just a dict and you could get only the field names simply by: TestClass.__fields__.keys() See model properties: https://pydantic-docs.helpmanual.io/usage/models/#model-properties
Question: I got the following waring from that code:  file = r'.\changed_activities.xlsx' with pd.ExcelWriter(file, engine='openpyxl', mode='a', if_sheet_exists='new') as writer: df.to_excel(writer, sheet_name=activity[0:30]) FutureWarning: Use of **kwargs is deprecated, use engine_kwargs instead. with pd.ExcelWriter(file,  What I can't seem to read from the documentation is which of my arguments I need to replace? the examples are e.g. this, but I don't see how that translates to my code. with pd.ExcelWriter( "path_to_file.xlsx", engine="openpyxl", mode="a", engine_kwargs={"keep_vba": True} ) as writer: df.to_excel(writer, sheet_name="Sheet2")  And maybe in a more open question: how/where can I understand something like that that in a general case?

Answer: Use pd.ExcelWriter('out.xlsx', engine='xlsxwriter', engine_kwargs={'options':{'strings_to_urls': False}})  Instead of pd.ExcelWriter('out.xlsx', engine='xlsxwriter', options={'strings_to_urls': False}})
Question: when I use docker-compose to install a fastapi project, I got AssertionError: jinja2 must be installed to use Jinja2Templates but when I use env to install it, that will be run well. my OS: Ubuntu18.04STL my requirements.txt: fastapi~=0.68.2 starlette==0.14.2 pydantic~=1.8.1 uvicorn~=0.12.3 SQLAlchemy~=1.4.23 # WSGI Werkzeug==1.0.1 pyjwt~=1.7.0 # async-exit-stack~=1.0.1 # async-generator~=1.10 jinja2~=2.11.2 # assert aiofiles is not None, "'aiofiles' must be installed to use FileResponse" aiofiles~=0.6.0 python-multipart~=0.0.5 requests~=2.25.0 pyyaml~=5.3.1 # html-builder==0.0.6 loguru~=0.5.3 apscheduler==3.7.0 pytest~=6.1.2 html2text==2020.1.16 mkdocs==1.2.1  Dockerfile FROM python:3.8 ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 WORKDIR /server COPY requirements.txt /server/ RUN pip install -r requirements.txt COPY . /server/  docker-compose.yml version: '3.7' services: figbox_api: build: context: . dockerfile: Dockerfile command: uvicorn app.main:app --port 8773 --host 0.0.0.0 --reload volumes: - .:/server ports: - 8773:8773  Do I need to provide some other information? Thanks

Answer: I had a same problem on heroku, the error comes from Jinja2 version 2.11.x and it run locally but not in Heroku. Just install latest version of jinja2 it will work fine in your case too. pip install Jinja2==3.1.2 or pip install Jinja2 --upgrade
Question: I am currently using a model from tf.keras.applications for training. And a data augmentation layer along with it. Wierdly, after I import the model from applications, the augmentation layer does not work. The augmentation layer does work before I import it. What is going on? Also, this has only started happening recently after the new version of TF 2.8.0 was released. Before it was working all fine. The code for the augmentation layer is data_augmentation = tf.keras.Sequential([ tf.keras.layers.RandomFlip("horizontal_and_vertical"), tf.keras.layers.RandomRotation(0.5), ])  And I am importing the model using base_model = tf.keras.applications.MobileNetV3Small( input_shape=(75, 50, 3), alpha=1.0, weights='imagenet', pooling='avg', include_top=False, dropout_rate=0.1, include_preprocessing=False)  Please help me understand what is going on. You can reproduce the code here on this notebook https://colab.research.google.com/drive/13Jd3l2CxbvIWQv3Y7CtryOdrv2IdKNxD?usp=sharing

Answer: I noticed same issue with tf 2.8. It can be solved by add training =True , when you test the augmentation layer: aug = data_augmentation(image,training=True)  The reason is that the augmentation layer behaves differently in training and predicting (inference), i.e. it will do augmentation in training but do nothing in predicting. Ideally, the layer should set the training= argument smartly according the situation. Apparently, it is not smart in above code: it does not know your intention is to test the layer. But I think you should still leave training argument as default when you build the full model, letting the augmentation layer do the job.
Question: Having a simple yet confusing issue: a package I added with poetry fails to import when I try to use it in a module. Steps taken:  poetry add sendgrid In a module, import sendgrid Error: Import "sendgrid" could not be resolved PylancereportMissingImports  Troubleshooting I've tried:  I checked my project's poetry venv dir, and sendgrid is there: 'C:\\Users\\xyz123\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\nameofproject-py3.10\\lib\\site-packages' Also checked sys.path(); the path to that site-packages dir is listed Running poetry install gives me the response No dependencies to install or update both the pyproject.toml and the poetry.lock files list sendgrid  What is going on?

Answer: Well, it turns out it's a matter of VSCode not playing nice and failing to recognize Poetry's virtual environment. I had to run the Python: Select Interpreter command and change the venv directory to the one my project is using, then it was able to recognize the installed packages. See here for more details on how to do that.