eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I currently work on a Python project, which I'd like to upload to the Microsoft Store in the future. As far as I am aware, in order to upload applications to the Microsoft Store, it is necessary that the application will be packed into the MSIX format. Now the question is - is it possible to pack a Python project into the MSIX format? I already tried two possible approaches   The first approach  I assumed that it will be much easier to pack an .exe file into an MSIX package. Since .py files require an interpreter in order to run, I managed to freeze the Python project into a standalone .exe runnable file - and it works pretty good. I found a useful tool made by Microsoft, which is supposed to pack .exe files under the MSIX format. The tool is MSIX Packaging Tool which is available at the Microsoft Store. I did manage to create an .msix file but I can't run in since Windows says that I have to sign the .exe first.   The second approach  I found out that it is possible to pack a project into an MSIX package, by using built-in tools inside Visual Studio 2019. So I managed to move my whole python project into Visual Studio, and follow the next steps in order to pack my project. The problem is that already in the early stages, when adding the reference to my python project, the next error occurs:   I'd love to know if you have any other possible approaches for packing a Python project into an MSIX package.

Answer:  Use PyInstaller or a similar tool to package your Python application. You can find more information on how to do this in the PyInstaller documentation.  Once you have the output from PyInstaller (either a single .exe file or the "dist" folder), you can use a program like AdvancedInstaller to create .msix files.  Note: If you're using the "dist" folder, don't add the folder itself to AdvancedInstaller. Instead, you'll find a folder inside the "dist" folder with the same name as your Python package or script. Add that folder to AdvancedInstaller.  In AdvancedInstaller, you can create shortcuts that point to the .exe file inside the folder.  PyInstaller: https://pyinstaller.org/en/stable/  AdvancedInstaller: https://www.advancedinstaller.com/  Creating an MSIX package using AdvancedInstaller: https://www.advancedinstaller.com/user-guide/tutorial-create-msix-package.html   Edit: You first approach actually works, you have to sign the exe indeed, I found this guide for signing the exe gratis https://adangel.org/2021/09/16/code-signing-lets-encrypt-github-pages/ Edit 2: Here's a useful link from Microsoft https://learn.microsoft.com/en-us/windows/apps/publish/publish-your-app/overview?pivots=store-installer-msi-exe
Question: When I try to export my Jupyter Notebook in pdf format in VSCode like this:   then I got this error:  Export failed. Please check the 'Jupyter' output panel for further details.  and jupyter output panel says:  [error] If you have not installed xelatex (TeX), you will need to do so before you can export to PDF. For further instructions, please see https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex. To avoid installing xelatex (TeX), you might want to try exporting to HTML and using your browser's "Print to PDF" feature.  so i tried to install MikTeX and update the required packages, but still I can't export Jupyter Notebooks in PDF format by VSCode! how can I fix this problem? Note That I know i can do it by convert it to HTML and then with ctrl+p try to save it as pdf! but I want to convert it to pdf in straight way!

Answer: Since I'm using conda venvs, I did these steps:  Activate conda venv using: conda activate <NAME_OF_VENV> in Anaconda prompt. Install nbconvert using conda install -c anaconda nbconvert  Now it's all okay, and I can export Jupyter notebooks in HTML and PDF format both. Update 11/17/2023 nbconvert is compatible with Python 3.8-3.11 based on the official doc.
Question: I'm looking for some sort of naming scheme for my virtual environments.  How do you usually name them? Is there naming convention for python virtual environments? 

Answer: If you are storing your environment inside the project folder some common names are env, venv, .env, .venv, but besides that, I don't think there are any common conventions. The official docs.python.org's tutorial on venv also suggests using .venv as the name.  A common directory location for a virtual environment is .venv. This name keeps the directory typically hidden in your shell and thus out of the way while giving it a name that explains why the directory exists. It also prevents clashing with .env environment variable definition files that some tooling supports. - docs.python.org/3/tutorial/venv.html
Question: How do I optimize for multiple metrics simultaneously inside the objective function of Optuna. For example, I am training an LGBM classifier and want to find the best hyperparameter set for all common classification metrics like F1, precision, recall, accuracy, AUC, etc. def objective(trial): # Train gbm = lgb.train(param, dtrain) preds = gbm.predict(X_test) pred_labels = np.rint(preds) # Calculate metrics accuracy = sklearn.metrics.accuracy_score(y_test, pred_labels) recall = metrics.recall_score(pred_labels, y_test) precision = metrics.precision_score(pred_labels, y_test) f1 = metrics.f1_score(pred_labels, y_test, pos_label=1) ...  How do I do it?

Answer: After defining the grid and fitting the model with these params and generate predictions, calculate all metrics you want to optimize for: def objective(trial): param_grid = {"n_estimators": trial.suggest_int("n_estimators", 2000, 10000, step=200)} clf = lgbm.LGBMClassifier(objective='binary', **param_grid) clf.fit(X_train, y_train) preds = clf.predict(X_valid) probs = clf.predict_proba(X_valid) # Metrics f1 = sklearn.metrics.f1_score(y_valid, press) accuracy = ... precision = ... recall = ... logloss = ...  and return them in the order you want: def objective(trial): ... return f1, logloss, accuracy, precision, recall  Then, in the study object, specify whether you want to minimize or maximize each metric to directions like so: study = optuna.create_study(directions=['maximize', 'minimize', 'maximize', 'maximize', 'maximize']) study.optimize(objective, n_trials=100)  For more details, see Multi-objective Optimization with Optuna in the documentation.
Question: I'm trying to use Column Transformer with OneHotEncoder to transform my categorical data : A quick look at my data :  I want to do one-hot-encoding for 3 features : 'sex' , 'smoker' , 'region', so I use Column Transformer by scikit-learn. ( I don't want to want to seperate numerical one and categorical one than transform them seperately, I just want to perform them on a single dataset) My code :  cat_feature = X.select_dtypes(include = 'object') #select only categorical columns enc = ColumnTransformer([ ('one_hot_encoder' , OneHotEncoder() , cat_feature ) ] , remainder = 'passthrough') X_transformed = enc.fit_transform(X) # transformed version of original data  My problem is that, X_transformed is then removed all the feature names which is little bit confusing for me to debug :  So is there anyway to retain my columns' names after doing this transformation? I want to incorporate this transformer into a pipeline so I can't use pd.get_dummies. Thank you!!

Answer: User will have to write custom Transformer which does passthrough and supports get_feature_names Steps:  Custom Transformer which will return pass through columns names via get_feature_names Dont use  remainder = 'passthrough' but rather use our custom Transformer  Use enc.get_feature_names() to get the feature list. Sample: from sklearn.base import BaseEstimator df = pd.DataFrame({ 'age': [1,2,3,4], 'sex': ['male', 'female']*2, 'bmi': [1.1,2.2,3.3,4.4], 'children': [1]*4, 'smoker': ['yes', 'no']*2 }) cat_features = df.select_dtypes(include = 'object').columns passthrough_features = [c for c in df.columns if c not in cat_features] class PassthroughTransformer(BaseEstimator): def fit(self, X, y = None): self.cols = X.columns return self def transform(self, X, y = None): self.cols = X.columns return X.values def get_feature_names(self): return self.cols enc = ColumnTransformer([ ('1hot' , OneHotEncoder() , cat_features ), ('pass' , PassthroughTransformer(), passthrough_features)]) X_transformed = enc.fit_transform(df) pd.DataFrame(X_transformed, columns=enc.get_feature_names())  Output:  1hot__x0_female 1hot__x0_male 1hot__x1_no 1hot__x1_yes pass__age pass__bmi pass__children 0 0.0 1.0 0.0 1.0 1.0 1.1 1.0 1 1.0 0.0 1.0 0.0 2.0 2.2 1.0 2 0.0 1.0 0.0 1.0 3.0 3.3 1.0 3 1.0 0.0 1.0 0.0 4.0 4.4 1.0
Question: While analyzing a code, I've stumbled upon the following snippet: msk = np.random.rand(len(df)) < 0.8  Variables "msk" and "df" are irrelevant for my question. After doing some research I think this usage is also related to "random" class as well. It gives True with 80% chance and False with 20% chance on random elements. It is done for masking. I understand why it is used but I don't understand how it works. Isn't random method supposed to give float numbers? Why are there boolean statements when we put the method in an interval?

Answer: np.random.rand(len(df)) returns an array of uniform random numbers between 0 and 1, np.random.rand(len(df)) < 0.8 will transform it into an array of booleans based on the condition. As there is a 80% chance to be below 0.8, there is 80% of True values. A more explicit approach would be to use numpy.random.choice: np.random.choice([True, False], p=[0.8, 0.2], size=len(df))  An even better approach, if your goal is to subset a dataframe, would be to use: df.sample(frac=0.8)  how to split a dataframe 0.8/0.2: df1 = df.sample(frac=0.8) df2 = df.drop(df1.index)
Question: I'm looking for Info on how to structure a Python package that wraps an extension module written in Rust, where both languages are mixed. I'm using pyO3 for FFI but can't seem to find an example on how to do this. To be specific: my rust library exposes a type that is later wrapped by a python class. Only the python class should be exposed for later users and the package should be structured, such that it can be pushed to PyPI. For example: On the rust side #[pyclass] pub struct Point { x: f64, y: f64 } #[pymethods] impl Point { #[new] pub fn new(x: f64, y: f64) -> Self { Self{x, y} } }  and on the python side from ??? import Point class Points: points: List[Point] def __init__(self, points: List[Tuple[float, float]]): self.points = [] for point in points: x, y = point self.points.append(Point(x, y))  I would be thankful for any Infos, Sources, Examples etc.!

Answer: I found a way to do this using Maturin. So, in case anyone else is trying to find out how to do this, here's one way. The project needs to have the following structure: my_project ├── Cargo.toml ├── my_project │ ├── __init__.py │ └── sum.py └── src └── lib.rs  Cargo.toml can be: [package] name = "my_project" version = "0.1.0" edition = "2018" [lib] name = "my_project" crate-type = ["cdylib"] [dependencies.pyo3] version = "0.14.5" features = ["extension-module"]  One example for lib.rs would be: use pyo3::prelude::*; #[pyfunction] fn sum_as_string(a: usize, b: usize) -> PyResult<String> { Ok((a + b).to_string()) } #[pymodule] fn my_project(_py: Python, m: &PyModule) -> PyResult<()> { m.add_function(wrap_pyfunction!(sum_as_string, m)?)?; Ok(()) }  Now in sum.py the function can be accessed (after using maturin develop during development, and when publishing automatically after maturin build): from .my_project import sum_as_string class Sum: sum: str def __init__(self, lhs: int, rhs: int): self.sum = sum_as_string(lhs, rhs)  The _ init _.py file can for example only expose the Sum class: from .sum import Sum
Question: When I development some socket.io service in python environment by using python-socketio and gunicorn, I meet an issue here. I am using Mac OS X and I am using python 3.7. Environment setting $ pip install python-socketio $ pip install gunicorn server-side code app.py import socketio sio = socketio.Server() app = socketio.WSGIApp(sio, static_files = { '/': './socketio-client.html' }) @sio.event def connect(sid, environ): print(sid, 'connected') @sio.event def disconnect(sid): print(sid, 'disconnected')  client-side code socketio-client.html <html> <head> <title>Socket.IO Demo</title> <script src="https://cdn.socket.io/3.1.3/socket.io.min.js" integrity="sha384-cPwlPLvBTa3sKAgddT6krw0cJat7egBga3DJepJyrLl4Q9/5WLra3rrnMcyTyOnh" crossorigin="anonymous"></script> </head> <body> <h1>Socket.IO Demo</h1> <script> const sio = io(); sio.on('connect', () => { console.log('connected'); }); sio.on('disconnect', () => { console.log('disconnected'); }); </script> </body> </html>  Start program I put the files in the same folder. And start it by following the command. $ gunicorn --thread 50 app:app  I use the browser to open http://localhost:8000 and that works. Issue But there is an issue with the server  The WebSocket transport is not available, you must install a WebSocket server that is compatible with your async mode to enable it. See the documentation for details. (further occurrences of this error will be logged with level INFO)  I try to do $ gunicorn --log-level INFO --thread 50 app:app But I still cannot get helpful information from INFO log-level. The code can keep going to run but the message shows me to install WebSocket server and I don't know which kind of package I need to install for this case of python-socketio or gunicorn. What kind of package I missed to install? It's my first time to use python-socketio and gunicorn. What should I do next? Thanks for your help.

Answer: It just needs to install more packages here. $ pip install gevent-websocket $ pip install eventlet And then $ gunicorn --thread 50 app:app  Update 1: If the server-side need to active emit to client side, it will need this environment. Because this command $ gunicorn --thread 50 app:app cannot support this situation. The worked environment should be set by following. $ pip install eventlet==0.30.2 $ gunicorn -k eventlet -w 1 app:app
Question: I was looking through my pip list and cleaning all my third party modules when I came across a module named 'distutils_hack'. I don't remember installing this, is this something I should be concerned about? The version I was using was Python 3.9. Thanks

Answer: It's used by setuptools to replace the stdlib distutils with setuptools' bundled distutils library. Quoting ncoghlan from pypa/setuptools#417 on why this is necessary:  While CPython as a whole has many contributors, we don't have many folks contributing to distutils any more - folks need build tools that let them target currently deployed versions of Python, which setuptools provides, but distutils really doesn't. So the challenge for pypa/packaging-problems#127 is as follows:  assume that in some future version of Python (3.9? 3.10? 4.0? We dunno yet) distutils is no longer in the standard library per se, but is instead bundled the way we bundle pip (i.e. by installing it from a wheel at Python installation time). eventually, for new Python versions (where distutils isn't in the standard library), we'd like the "real" distutils code to live in setuptools, and the code you get when you do import distutils to just be a backward compatibility shim that aliases the setuptools components into the right place however, for old Python versions (where distutils is still in the standard library), then setuptools will still need to do its monkeypatching magic to allow plain distutils projects to emit the installation database metadata and to be built as wheel archives  The scope of the eventual distutils-compat shim probably wouldn't need to be the full distutils API, but it wouldn't be the empty set either. So @jaraco's suggestion sounds like a reasonable starting point to me: give setuptools its own pre-patched copy of distutils so it can work on a distutils-free copy of Python, without providing an importable distutils facade. Building a suitable facade (with a setuptools dependency) would then be part of removing distutils from the standard library.  Here is where you can find the module's source code: https://github.com/pypa/setuptools/blob/main/_distutils_hack/__init__.py
Question: Give something as follows: import importlib module_path = "mod" mod = importlib.import_module(module_path, package=None) print(mod.Foo.Bar.x)  where mod.py is: class Foo: class Bar: x = 1  mypy file.py --strict raises the following error: file.py:7: error: Module has no attribute "Foo" [attr-defined]  I'm wondering how one is supposed to go about type-hinting this, or if this is something which would typically just be ignored with # type: ignore[attr-defined] (assuming that the code is necessary, and the only options are type-hinting or ignoring the type-hint)?  Why I am using importlib in this situation The way that importlib is being used is that there's some path: x.y.<changes>.z  Where <changes> is dynamic, but the others are fixed. I'm confident that the module will contain the attributes which are being called, but due to <changes>, importlib is used for the import. Which might be summarised as:  I do not know precisely which module I will be importing, but I know it will have a class Foo in it. 

Answer: As was alluded to by @MisterMiyagi in the comments, I think the solution here is to use structural, rather than nominal, subtyping. Nominal subtyping is where we use direct class inheritance to define type relationships. For example, collections.Counter is a subtype of dict because it directly inherits from dict. Structural subtyping, however, is where we define types based on certain properties a class has or certain behaviours it displays. int is a subtype of typing.SupportsFloat not because it directly inherits from SupportsFloat (it doesn't), but because SupportsFloat is defined as a certain interface, and int satisfies that interface. When type-hinting, we can define structural types using typing.Protocol. You could satisfy MyPy in this situation like this: import importlib from typing import cast, Protocol class BarProto(Protocol): x: int class FooProto(Protocol): Bar: type[BarProto] class ModProto(Protocol): Foo: type[FooProto] module_path = "mod" mod = cast(ModProto, importlib.import_module(module_path, package=None)) print(mod.Foo.Bar.x) reveal_type(mod) reveal_type(mod.Foo) reveal_type(mod.Foo.Bar) reveal_type(mod.Foo.Bar.x)  We've defined several interfaces here:  BarProto: in order to satisfy this interface, a type has to have an attribute x that's of type int. FooProto: in order to satisfy this interface, a type has to have an attribute Bar that is a class of which instances satisfy the BarProto protocol. ModProto: in order to satisfy this interface, a type has to have an attribute Foo that is a class of which instances satisfy the FooProto protocol.  Then, when importing the module, we use typing.cast to assert to the type-checker that the module we're importing satisfies the ModProto protocol. Run it through MyPy, and it informs us it has inferred the following types: main.py:18: note: Revealed type is "__main__.ModProto" main.py:19: note: Revealed type is "Type[__main__.FooProto]" main.py:20: note: Revealed type is "Type[__main__.BarProto]" main.py:21: note: Revealed type is "builtins.int"  Read more about structural subtyping in python here and here.
Question: Context I have a test class where all my tests inherit from. It cant run by itself as it really doesnt contain any setup info I wanted to add a test which is executed by ALL tests (adding it to the baseclass seems logical) But now I notice the basetestclass( => Foo) which I import is being detected as a test itself and runs and is visible in the reports Code the base class in base.py from unittest import TestCase class Foo(TestCase): @classmethod def setUpClass(cls): # prepare the generic setup stuff based on what is defined in the child class print("setupclass Foo done") def test_run_in_all_inherited_tests(self): print("fooBar") assert True  the real test in test_something.py from base import Foo # <= This is being detected as a testclass object and thus will be executed class TestFoo(Foo): @classmethod def setUpClass(cls): # define specific test setup super().setUpClass() print("setup TestFoo done") def test_pass(self): pass def test_assert(self): assert False  This triggers a testrun of the imported Foo  The Question How can I import Foo without that its being detected as a 'test' If I remove the test to run in all tests all is fine. Adding @nottest decorator to Foo wont work since then also all inherited classes are defined nottest. It needs to run on nose, pytest and unittest testrunners I noticed if I changed the import statement like below that it also works. But that would mean adjusting a few hundreds of testfiles in different repos. (I'd like to avoid that) import base class TestFoo(base.Foo): 

Answer: The key to the answer seems to be that each test has an attribute __test__ which is set to True when it is a test. Setting it to False when the class should not be a test will then let the test collector ignore this class. The answer assumes I can only do changes in the base.py In python 3.9 classmethod and property decorators can be combined so I wrote a separate answer for that answer for < py3.9 the base class in base.py from unittest import TestCase class MetaFoo(type): @property def __test__(cls): return cls != Foo class Foo(TestCase, metaclass=MetaFoo): @classmethod def setUpClass(cls): # prepare the generic setup stuff based on what is defined in the child class print("setupclass Foo done") def test_run_in_all_inherited_tests(self): print("fooBar") assert True  answer for >= py3.9 the base class in base.py from unittest import TestCase class Foo(TestCase): @classmethod @property def __test__(cls): return cls != Foo @classmethod def setUpClass(cls): # prepare the generic setup stuff based on what is defined in the child class print("setupclass Foo done") def test_run_in_all_inherited_tests(self): print("fooBar") assert True  the actual test test_something.py from base import Foo # <= This will not be detected as a test anymore as __test__ returns False class TestFoo(Foo): @classmethod def setUpClass(cls): # define specific test setup super().setUpClass() print("setup TestFoo done") def test_pass(self): pass def test_assert(self): assert False  This doesnt trigger a testrun of the imported Foo anymore