eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to show a tree visualisation using plot_tree, but it shows a chunk of text instead: from sklearn.tree import plot_tree plot_tree(t)  (where t is an instance of DecisionTreeClassifier) This is the output: [Text(464.99999999999994, 831.6, 'X[3] <= 0.8\nentropy = 1.581\nsamples = 120\nvalue = [39, 37, 44]'), Text(393.46153846153845, 646.8, 'entropy = 0.0\nsamples = 39\nvalue = [39, 0, 0]'),  and so on and so forth. How do I make it show the visual tree instead? I'm using Jupyter 6.4.1 and I already imported matplotlib earlier in the code. Thanks!

Answer: In my case, it works with a simple "show": plot_tree(t) plt.show()
Question: Is there an easy way to fix this error: Missing Python executable 'python3', defaulting to 'C:\Users\user1\Anaconda3\Lib\site-packages\pyspark\bin\..' for SPARK_HOME environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON environment variable to detect SPARK_HOME safely.  Would I have to modify the PATH system variable? Or export/create the environment variables PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON? I have Python 3.8.8.

Answer: you need to add an environment variable called SPARK_HOME: this variable contain the path to the installed pyspark library . In my case , pyspark is installed under my home directory, so this is the content of the variable : SPARK_HOME=/home/zied/.local/lib/python3.8/site-packages/pyspark  also you need another variable called PYSPARK_PYTHON which have the python version you are using like this : PYSPARK_PYTHON=python3.8  EDIT: for windows use PYSPARK_PYTHON=python
Question: First, I tried to load using: (X_train, y_train), (X_test, y_test) = datasets.cifar10.load_data()  But it gave an error: Exception: URL fetch failure on https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz: None -- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1125)  So I manually downloaded the dataset and put it in C:\Users\SAHAN\.keras\datasets and renamed it to cifar-10-batches-py.tar.gz. But then it gives an error: PermissionError: [Errno 13] Permission denied: 'C:\\Users\\SAHAN\\.keras\\datasets\\cifar-10-batches-py.tar.gz'  How can I load this dataset?

Answer: I was having a similar CERTIFICATE_VERIFY_FAILED error downloading CIFAR-10. Putting this in my python file worked: import ssl ssl._create_default_https_context = ssl._create_unverified_context  Reference: https://programmerah.com/python-error-certificate-verify-failed-certificate-has-expired-40374/
Question: I use setuptools. In setup.cfg, I can define [options.package_data] myModule = '*.csv'  to make sure that data will be installed for my users. Can I achive the same with pyproject.toml instead?

Answer: Starting in setuptools version 61.0.0 there is beta support for this feature. See the documentation https://setuptools.pypa.io/en/stable/userguide/datafiles.html The syntax is [tool.setuptools.package-data] myModule = ["*.csv"]  Update 2024-01-12: This is no longer in beta. A handful of different mechanisms can enable for data inclusion. One mechanism is to ask for automatic discovery based on git. It should suffice to set the setuptools version and enable the -scm plugin. Now all files tracked by git will be considered data files. The relevant section is... [build-system] requires = [ "setuptools>=60", "setuptools-scm>=8.0"]  I produced a complete example and posted on github. https://github.com/el-hult/mypkg Read the full documentation for more options.
Question: Suppose I have the following function: def foo(df: pd.DataFrame) -> pd.DataFrame: x = df["x"] y = df["y"] df["xy"] = x * y return df  Is there a way I could hint that my function is accepting a data frame that must have the "x" and "y" column and that it will return a data frame with the "x", "y" and "xy" columns, instead of just a general data frame?

Answer: Okay, so, I'm not sure if this is the correct way of implementing it, but seems to work for me. If you see any mistakes or alternatives let me know and I can edit the response but my solution was basically creating a new class and implementing the __class_getitem__ method as seen in the Pep 560, this was my final code: from typing import List import pandas as pd GenericAlias = type(List[str]) class MyDataFrame(pd.DataFrame): __class_getitem__ = classmethod(GenericAlias) def foo(df: MyDataFrame[["x", "y"]]) -> MyDataFrame[["x", "y", "xy"]]: df["xy"] = df["x"] * df["y"] return df   Edit The Pandera library is a better alternative that I did not know about when I posted the question. import pandas as pd from pandera.typing import Series, DataFrame class InputSchema(pa.DataFrameModel): x: Series[float] y: Series[float] class ReturnSchema(pa.DataFrameModel): x: Series[float] y: Series[float] xy: Series[float] def foo(df: DataFrame[InputSchema]) -> DataFrame[ReturnSchema]: x = df['x'] y = df['y'] df['xy'] = x * y return df
Question: The new structural pattern matching feature in Python 3.10 is a very welcome feature. Is there a way to match inequalities using this statement? Prototype example: match a: case < 42: print('Less') case == 42: print('The answer') case > 42: print('Greater') 

Answer: You can use guards: match a: case _ if a < 42: print('Less') case _ if a == 42: print('The answer') case _ if a > 42: print('Greater')  Another option, without guards, using pure pattern matching: match [a < 42, a == 42]: case [True, False]: print('Less') case [_, True]: print('The answer') case [False, False]: print('Greater')
Question: When trying to install the python moduel pikepdf using pip, this error pops up: Building wheels for collected packages: pikepdf Building wheel for pikepdf (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for pikepdf (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [54 lines of output] ... creating build\temp.win-amd64-cpython-310\Release\src\qpdf "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DPOINTERHOLDER_TRANSITION=4 -IC:\Users\ME\AppData\Local\Temp\pip-build-env-dpc9ltd5\overlay\Lib\site-packages\pybind11\include "-IC:\Program Files\Python310\include" "-IC:\Program Files\Python310\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /EHsc /Tpsrc/qpdf\annotation.cpp /Fobuild\temp.win-amd64-cpython-310\Release\src/qpdf\annotation.obj /EHsc /bigobj /std:c++17 annotation.cpp src/qpdf\annotation.cpp(4): fatal error C1083: Cannot open include file: 'qpdf/Constants.h': No such file or directory error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pikepdf Failed to build pikepdf ERROR: Could not build wheels for pikepdf, which is required to install pyproject.toml-based projects  Creating the wheel fails due to a missing header file: src/qpdf\annotation.cpp(4): fatal error C1083: Cannot open include file: 'qpdf/Constants.h': No such file or directory  This is for pikepdf v6.0.0. My previous version was v4.0.1.post1, which worked fine. Is this something that can be remedied from my side?

Answer: Solution for Macbook, M1 brew install qpdf  After it use pip install pikepdf  Solution from https://github.com/pikepdf/pikepdf/issues/274
Question: When a project is specified only via pyproject.toml (i.e. no setup.{py,cfg} files), how can it be installed in editable mode via pip (i.e. python -m pip install -e .)? I tried both setuptools and poetry for the build system, but neither worked: [build-system] requires = ["setuptools", "wheel"] build-backend = "setuptools.build_meta"  [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"  I get the same error for both build systems: ERROR: Project file:///tmp/demo has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.  I'm using this inside a conda environment, the following is my version of setuptools and pip: $ conda list | grep setuptools setuptools 58.3.0 pypi_0 pypi $ python -m pip --version pip 21.3.1 

Answer: PEP 660 – Editable installs for pyproject.toml based builds defines how to build projects that only use pyproject.toml. Build tools must implement PEP 660 for editable installs to work. You need a front-end (such as pip ≥ 21.3) and a backend. The statuses of some popular backends are:  Setuptools implements PEP 660 as of version 64. Flit implements PEP 660 as of version 3.4. Poetry implements PEP 660 as of version 1.0.8.  Note: To be able to do an editable installation to your user site (pip install -e --user), you need a system installed setuptools v62.0.0 or newer.
Question: I am trying to use multiple cases in a function similar to the one shown below so that I can be able to execute multiple cases using match cases in python 3.10 def sayHi(name): match name: case ['Egide', 'Eric']: return f"Hi Mr {name}" case 'Egidia': return f"Hi Ms {name}" print(sayHi('Egide'))  This is just returning None instead of the message, even if I remove square brackets.

Answer: According to What’s New In Python 3.10, PEP 636, and the docs, you use a | between patterns: case 'Egide' | 'Eric':
Question: So I know that in my unit test I can mock a context manager open(), i.e.: with open('file_path', 'r') as stats:  mocked with with mock.patch('builtins.open', mock.mock_open(read_data=mock_json)):  but is there a way for me to only mock it for a specific file path? Or maybe some other way to ensure that the context manager gets called with the correct path in a unit test?

Answer: To mock open only for a specific path, you have to provide your own mock object that handles open differently, depending on the path. Assuming we have some function: def do_open(path): with open(path, "r") as f: return f.read()  where open shall be mocked to return a file with the content "bar" if path is "foo", but otherwise just work as usual, you could do something like this: from unittest import mock from my_module.do_open import do_open builtin_open = open # save the unpatched version def mock_open(*args, **kwargs): if args[0] == "foo": # mocked open for path "foo" return mock.mock_open(read_data="bar")(*args, **kwargs) # unpatched version for every other path return builtin_open(*args, **kwargs) @mock.patch("builtins.open", mock_open) def test_open(): assert do_open("foo") == "bar" assert do_open(__file__) != "bar"  If you don't want to save the original open in a global variable, you could also wrap that into a class: class MockOpen: builtin_open = open def open(self, *args, **kwargs): if args[0] == "foo": return mock.mock_open(read_data="bar")(*args, **kwargs) return self.builtin_open(*args, **kwargs) @mock.patch("builtins.open", MockOpen().open) def test_open(): ...
Question: I am trying to speed up some code using numba, but it is tough sledding. For example, the following function does not numba-fy, @jit(nopython=True) def returns(Ft, x, delta): T = len(x) rets = Ft[0:T - 1] * x[1:T] - delta * np.abs(Ft[1:T] - Ft[0:T - 1]) return np.concatenate([[0], rets])  because numba cannot find the signature of np.concatenate. Is there a canonical fix for this?

Answer: A bit late, but I hope still useful. Since you asked for the "canonical fix", I would like to explain why concatenate is a bad idea when working with arrays and especially if you indicate that you want to remove bottlenecks and therefore use the numba jit. An array is a continuous sequence of bytes in memory (numpy knows some tricks to change the order without copying by creating views, but that is another topic, see https://towardsdatascience.com/advanced-numpy-master-stride-tricks-with-25-illustrated-exercises-923a9393ab20). If you want to prepend the value x to an array of N elements, you will need to create a new array with N+1 elements, set the first value to x and copy the remaining part. As a side note, a similar argument holds for prepending items to a python list, which is the reason why collections.deque exists. Now, in your jit decorated function, you could hope that the compiler understands what you want to do, but writing compilers that always understands what you are trying to do is nearly impossible. Therefore, better be kind to the compiler and help out with the memory layout whenever you know the right choice. Thus, IMHO the "canonical fix" to your example code would be something like the following: @jit(nopython=True) def returns(Ft, x, delta): T = len(x) rets = np.empty_like(x) rets[0] = 0 rets[1:T] = Ft[0:T - 1] * x[1:T] - delta * np.abs(Ft[1:T] - Ft[0:T - 1]) return rets  In general, I agree with @Aaron's comment, meaning that you should always be as explicit as possible with input types to any function you call within jit decorated functions. In your case, ask yourself as a compiler "what is [[0], rets]?". Thinking in strict types, you see a list containing a list of an integer and an array of floating point (or complex) numbers. That is a challenging mixture of types for a compiler. Should the output become an array of integers or floats?
Question: I have a dataframe and did some feature engineering and now would like to change the column names. I know how to change them if I do a new assignment but I would like to do it with method chaining. I tried the below (the rename row) but it doesn't work. How could I write it so it works? df = pd.DataFrame({'ID':[1,2,2,3,3,3], 'date': ['2021-10-12','2021-10-16','2021-10-15','2021-10-10','2021-10-19','2021-10-01'], 'location':['up','up','down','up','up','down'], 'code':[False, False, False, True, False, False]}) df = (df .assign(date = lambda x: pd.to_datetime(x.date)) .assign(entries_per_ID = lambda x: x.groupby('ID').ID.transform('size')) .pivot_table(values=['entries_per_ID'], index=['ID','date','code'], columns=['location'], aggfunc=np.max) .reset_index() #.rename(columns=lambda x: dict(zip(x.columns, ['_'.join(col).strip() if col[1]!='' else col[0] for col in x.columns.values]))) )  This here works, but that's not how I would like to write it. df.columns = ['_'.join(col).strip() if col[1]!='' else col[0] for col in df.columns.values ] 

Answer: Renaming columns in a chain Use set_axis along axis=1: df.set_axis(['foo', 'bar', 'baz'], axis=1)   Using groupby, pivot, melt, etc If the new columns depend on some earlier step in the chain, combine set_axis with pipe. For example, to capitalize pivoted columns in a chain:  We cannot directly chain set_axis: # does NOT work since df.columns are the original columns, not pivoted columns df.pivot(...).set_axis(df.columns.str.upper(), axis=1))   But we can pipe the pivoted result into set_axis: # does work since we've piped the pivoted df df.pivot(...).pipe(lambda piv: piv.set_axis(piv.columns.str.upper(), axis=1))) # ^ ^ ^     OP's example Since OP has created a pivot_table and wants to conditionally collapse those pivoted MultiIndex, we pipe the pivot_table into the list comprehension: (df.assign(date=pd.to_datetime(df.date)) .assign(entries_per_ID=df.groupby('ID').ID.transform('size')) .pivot_table(index=['ID', 'date', 'code'], columns='location', values='entries_per_ID', aggfunc='max') .reset_index() .pipe(lambda piv: piv.set_axis(['_'.join(col).strip() if col[1] else col[0] for col in piv.columns], axis=1))) # ID date code entries_per_ID_down entries_per_ID_up # 0 1 2021-10-12 False NaN 1.0 # 1 2 2021-10-15 False 2.0 NaN # 2 2 2021-10-16 False NaN 2.0 # 3 3 2021-10-01 False 3.0 NaN # 4 3 2021-10-10 True NaN 3.0 # 5 3 2021-10-19 False NaN 3.0
Question: I write code in Python in VS code. If I add comment before function and hit save button, VS code adds two empty lines: # comment def MyMethod(): return 0  In settings I see that I use autopep8 formatter:  I wasnt able to find what causes this annoying issue. Maybe I can configure settings somewhere?

Answer: Comment documenting function behavior should be placed inside function, just below signature. If it is not describing function (is placed outside function) it should have those blank lines. Don't mess with language's conventions, it's very bad idea. Edit, Further clarification: """Module-level comment """ def function(): """Function level comment. There are multiple conventions explaining how a comment's body should be formed. """ return 0
Question: Coming from C/C++ background, I am aware of coding standards that apply for Safety Critical applications (like the classic trio Medical-Automotive-Aerospace) in the context of embedded systems , such as MISRA, SEI CERT, Barr etc. Skipping the question if it should or if it is applicable as a language, I want to create Python applications for embedded systems that -even vaguely- follow some safety standard, but couldn't find any by searching, except from generic Python coding standards (like PEP8) Is there a Python coding guideline that specificallly apply to safety-critical systems ?

Answer: Top layer safety standards for "functional safety" like IEC 61508 (industrial), ISO 26262 (automotive) or DO-178 (aerospace) etc come with a software part (for example IEC 61508-3), where they list a number of suitable programming languages. These are exclusively old languages proven in use for a long time, where all flaws and poorly-defined behavior is regarded as well-known and execution can be regarded as predictable. In practice, for the highest safety levels it means that you are pretty much restricted to C with safe subset (MISRA C) or Ada with safe subset (SPARK). A bunch of other old languages like Modula-2, Pascal and Fortran are also mentioned, but the tool support for these in the context of modern safety MCUs is non-existent. As is support for Python for such MCUs. Languages like Python and C++ are not even mentioned for the lowest safety levels, so between the lines they are dismissed as entirely unsuitable. Even less so than pure assembler, which is actually mentioned as something that may used for the lower safety levels.
Question: We are writing a web service using FastAPI that is going to be hosted in Kubernetes. For auditing purposes, we need to save the raw JSON body of the request/response for specific routes. The body size of both request and response JSON is about 1MB, and preferably, this should not impact the response time. How can we do that?

Answer: You may try to customize APIRouter like in FastAPI official documentation: import time from typing import Callable from fastapi import APIRouter, FastAPI, Request, Response from fastapi.routing import APIRoute class TimedRoute(APIRoute): def get_route_handler(self) -> Callable: original_route_handler = super().get_route_handler() async def custom_route_handler(request: Request) -> Response: before = time.time() response: Response = await original_route_handler(request) duration = time.time() - before response.headers["X-Response-Time"] = str(duration) print(f"route duration: {duration}") print(f"route response: {response}") print(f"route response headers: {response.headers}") return response return custom_route_handler app = FastAPI() router = APIRouter(route_class=TimedRoute) @app.get("/") async def not_timed(): return {"message": "Not timed"} @router.get("/timed") async def timed(): return {"message": "It's the time of my life"} app.include_router(router)
Question: I have a FastAPI application that I am running on port 30000 using Uvicorn programmatically. Now I want to run the same application on port 8443 too. The same application needs to run on both these ports. How can I do this within the Python code? Minimum Reproducible code: from fastapi import FastAPI import uvicorn app = FastAPI() @app.get("/healthcheck/") def healthcheck(): return 'Health - OK' if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", port=30000)  I want to something like if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", ports=[30000,8443])  Explanation: My application will be running on my organizations Azure Kubernetes Service. Apps running on port 30000 are reserved for Internal HTTP traffic and apps running on 8443 are mapped to 443 of Kubernetes Service to be exposed to external traffic. Further Details: I will be creating a Docker Container out of this application and the idea is to include CMD ["python3", "app.py"]  at the end to run the application. I am looking for a solution that would either provide a way to change the python code ( uvicorn.run(app, host="0.0.0.0", ports=[30000,8443]) ) or a change to the CMD command in the Dockerfile like This GitHub Issue Comment - gunicorn -k uvicorn.workers.UvicornWorker -w 1 --bind ip1:port1 --bind ip2:port2 --bind ip3:port3 

Answer: The following solution worked for me. It runs one gunicorn process in the background and then another process to bind it to two ports. One of them will use HTTP and one can use HTTPS. Dockerfile: FROM python:3.7 WORKDIR /app COPY requirements.txt . RUN pip3 install -r requirements.txt COPY . . ENTRYPOINT ./docker-starter.sh EXPOSE 30000 8443  docker-starter.sh: gunicorn -k uvicorn.workers.UvicornWorker -w 3 -b 0.0.0.0:30000 -t 360 --reload --access-logfile - app:app & gunicorn --access-logfile - -k --ca_certs ca_certs.txt uvicorn.workers.UvicornWorker -w 3 -b 0.0.0.0:8443 -t 360 --reload --access-logfile - app:app  The python app can remain minimal: from fastapi import FastAPI import uvicorn app = FastAPI() @app.get("/healthcheck/") def healthcheck(): return 'Health - OK' if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0")