eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've got a function that takes an arbitrary amount of lists (or any iterables, for that matter) and sorts them as one. The code looks like this: def sort_as_one(*args): return zip(*sorted(zip(*args))) def main(): list1 = [3, 1, 2, 4] list2 = ["a", "b", "d", "e"] list3 = [True, False, True, False] result = sort_as_one(list1, list2, list3) # <zip object at ...> print(result) list1, list2, list3 = result print(list1, list2, list3) if __name__ == "__main__": main()  How can I accurately type hint the function output?

Answer: A zip object is an iterator - it follows the iterator protocol. Idiomatically, you would probably just typing hint it as such. In this case, you want to type hint it as a generic using a type variable: import typing T = typing.TypeVar("T") def sort_as_one(*args: T) -> typing.Iterator[T]: return zip(*sorted(zip(*args)))  Note, if you are using variadic arguments, you have to only accept a single type. In this case, the best you can probably do in your case is use Any instead of T. But you should consider using only a function like the above in your code if you want to be able to use it with static type checkers.
Question: I have a column in my dataframe that contains nan values and int values. The original dType was float64, but I was trying to change it to int6, and change nan values to np.nan. now I get this error: TypeError: float() argument must be a string or a number, not 'NAType' when trying to run imputation on it. In the following table, column is similar to "age" data = {'name': ['Alex', 'Ben', 'Marry','Alex', 'Ben', 'Marry'], 'job': ['teacher', 'doctor', 'engineer','teacher', 'doctor', 'engineer'], 'age': [27, 32, 78,27, 32, 78], 'weight': [160, 209, 130,164, 206, 132], 'date': ['6-12-2022', '6-12-2022', '6-12-2022','6-13-2022', '6-13-2022', '6-13-2022'] } df = pd.DataFrame(data) df |name |job |age|weight |date |---|-------|-----------|---|-------|-------- |0 |Alex |teacher |27 |160 |6-12-2022 |1 |Ben |doctor |32 |209 |6-12-2022 |2 |Marry |engineer |78 |130 |6-12-2022 |3 |Alex |teacher |27 |164 |6-13-2022 |4 |Ben |doctor |32 |206 |6-13-2022 |5 |Marry |engineer |78 |132 |6-13-2022 |6 |Alex |teacher |NaN|NaN |6-14-2022 |7 |Ben |doctor |NaN|NaN |6-14-2022 |8 |Marry |engineer |NaN|NaN |6-14-2022  and this is what I tried: df['age']=df['age'].astype( dtype={'age': pd.Int8Dtype()}) df.loc[df.age== '<NA>', 'age']=np.nan  Is there any way to change float64 to smaller datatype without causing this issue? Please advise, thanks

Answer: Use df['age'] = df['age'].astype(dtype='Int64')  with extension datatype Int64 (with a capitalized I) rather than the default dtype which is int64 (lower case i). The latter throws an IntCastingNaNError while the former works smoothly. This functionality was added to Pandas 0.24 and mentioned in this thread.
Question: Which magic method method does hasattr call? getattr(__o, name) can also be called as __o.__getattr__(name) setattr(__o, name) can also be called as __o.__setattr__(name) But what is the equivalent for hasattr? I know the associated magic method for the in keyword is __contains__.

Answer: There is no specific dunder method for hasattr(). It's essentially equivalent to: def hasattr(object, name): try: getattr(object, name) return True except AttributeError: return False  So it's dependent on the same dunder methods used by getattr().
Question: Consider the following enum class: from enum import Enum class Namespace: class StockAPI(Enum): ITEMS = "{url}/items" INVENTORY = "{url}/inventory" class CustomerAPI(Enum): USERS = "{url}/users" PURCHASES = "{url}/purchases" def __init__(self, url): self.url = url  I am trying to make url a dynamic value for each enum class. What can I do here so that I can call some enum class in one of the following ways:  Namespace.StockAPI.ITEMS.value would return http://localhost/items? Namespace(url="http://localhost").StockAPI.ITEMS.value would also return http://localhost/items  Is this possible to do without doing variable interpolation each time I access each enum property? Could factory pattern be of any help here?

Answer: I did it the following way, while keeping the inner enum classes: from enum import Enum class Namespace: class StockAPI(Enum): ITEMS = "{url}/items" INVENTORY = "{url}/inventory" class CustomerAPI(Enum): USERS = "{url}/users" PURCHASES = "{url}/purchases" def __init__(self, url: str): attrs = (getattr(self, attr) for attr in dir(self)) enums = (a for a in attrs if isinstance(a, type) and issubclass(a, Enum)) for enum in enums: kv = {e.name: e.value.format(url=url) for e in enum} setattr(self, enum.__name__, Enum(enum.__name__, kv)) print(Namespace(url="http://test.com").StockAPI.ITEMS.value) # http://test.com/items
Question: I have a large multi-file Python application I'd like to document graphically. But first, I made a small "dummy" app to test out different UML packages. (Note: I do have graphviz installed and in the path). Here's my "dummy" code: class User: def __init__(self, level_security=0): self.level_security = level_security def level_security_increment(level_security): level_security += 1 return level_security def printNice(aValue): if type(aValue) != "str": print(str(aValue)) else: print(aValue) return def main(): printNice("Process start") for x in range(0,9): printNice(x) myUser = User(3) printNice(f"Sec lvl: {myUser.level_security}") printNice("Process_done") if __name__ == "__main__": main()  Here are the different pyreverse command-line codes I've used to get varying charts. I'll post the chart that came closest to what I want below these codes: pyreverse test.py -S -m y -A -o png pyreverse -ASmn test.py -b -o vdx pyreverse -ASmy test.py -o emf pyreverse -AS -m y test.py -o emf pyreverse -p test.py -o emf pyreverse -AS -b test.py -o emf pyreverse -ASmy test.py -b -o png <-- closest  Now here is what that last one you see above produces:  Finally, in case it is not clear, I'll reiterate what I want it to show: Classes, Definitions (functions), and - if possible - even variables. But I'd be happy for now to get just Classes and Definitions (functions). To be clear: I want to see function names. Is it as simple as adding/removing a switch to the pyreverse command? Or is there some package I need to add to my "dummy" code?

Answer: pyreverse aims to produce a class diagram. It will show you classes, and non-filtered class members (see option -f), as well as associations that can be detected. In this regard, the diagram seems complete. Instances (objects) at top level are not part of a class diagram. This is why pyreverse doesn't show them. Free standing functions do not appear in class diagrams either as they are not classes. There is no consensus about what a free standing function should be in UML. They could be seen as instances of a more general function class (of all the functions with the same signature), or they could be considered as a specific functor class. As no UML rule is defined, pyrevere doesn't show them either. If you want pyreverse to detect them, you should rewrite them as a functor class.
Question: Pytorch's docs on Dataloaders only say, in the code def prepare_data(self): # download ...  and def setup(self, stage: Optional[str] = None): # Assign train/val datasets for use in dataloaders  Please explain the intended separation between prepare_data and setup, what callbacks may occur between them, and why put something in one over the other.

Answer: If you look at the pseudo for the Trainer.fit function provided in the documentation page of LightningModule at § Hooks, you can read: def fit(self): if global_rank == 0: # prepare data is called on GLOBAL_ZERO only prepare_data() ## <-- prepare_data configure_callbacks() with parallel(devices): # devices can be GPUs, TPUs, ... train_on_device(model) def train_on_device(model): # called PER DEVICE on_fit_start() setup("fit") ## <-- setup configure_optimizers() # the sanity check runs here on_train_start() for epoch in epochs: fit_loop() on_train_end() on_fit_end() teardown("fit")  You can see prepare_data being called only for global_rank == 0, i.e. it is only called by a single processor. It turns out you can read from the documentation description of prepare_data:  LightningModule.prepare_data() Use this to download and prepare data. Downloading and saving data with multiple processes (distributed settings) will result in corrupted data. Lightning ensures this method is called only within a single process, so you can safely add your downloading logic within.  Whereas setup is called on all processes as you can read from the pseudo-code above as well as its documentation description:  LightningModule.setup(stage=None)Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.
Question: I am new to pyspark and using Dataframes what I am trying to do is get the subset of all the columns with Null value(s). Most examples I see online show me a filter function on a specific column. Is it possible to filter the entire data frame and show all the rows that contain at least 1 null value?

Answer: If you don't care about which columns are null, you can use a loop to create a filtering condition: from pyspark.sql import SparkSession from pyspark.sql import functions as func q1_df = spark\ .createDataFrame([(None, 1, 2), (3, None, 4), (5, 6, None), (7, 8, 9)], ['a', 'b', 'c']) q1_df.show(5, False) +----+----+----+ |a |b |c | +----+----+----+ |null|1 |2 | |3 |null|4 | |5 |6 |null| |7 |8 |9 | +----+----+----+ condition = (func.lit(False)) for col in q1_df.columns: condition = condition | (func.col(col).isNull()) q1_df.filter(condition).show(3, False) +----+----+----+ |a |b |c | +----+----+----+ |null|1 |2 | |3 |null|4 | |5 |6 |null| +----+----+----+  As you're finding the row that any one column is null, you can use the OR condition.  Edit on: 2022-08-01 The reason why I first declare condition as func.lit(False) is just for the simplification of my coding, just want to create a "base" condition. In fact, this filter doesn't have any usage in this filtering. When you check the condition, you will see: Column<'(((false OR (a IS NULL)) OR (b IS NULL)) OR (c IS NULL))'>  In fact you can use other method to create the condition. For example: for idx, col in enumerate(q1_df.columns): if idx == 0: condition = (func.col(col).isNull()) else: condition = condition | (func.col(col).isNull()) condition Column<'(((a IS NULL) OR (b IS NULL)) OR (c IS NULL))'>  Alternatively, if you want to filter out the row that BOTH not null in all columns, in my coding, I would: condition = (func.lit(True)) for col in q1_df.columns: condition = condition & (func.col(col).isNotNull())  As long as you can create all the filtering condition, you can eliminate the func.lit(False). Just to remind that if you create the "base" condition like me, please don't use the python built-in bool type like below since they are not the same type (boolean vs spark column): condition = False for col in q1_df.columns: condition = condition | (func.col(col).isNull())
Question: I'm trying to follow the fine tuning guide for Openai here. I ran: pip install --upgrade openai  Which install without any errors. But even after restarting my terminal, i still get zsh: command not found: openai  Here is the output of echo $PATH: /bin:/usr/bin:/usr/local/bin:/Users/nickrose/Downloads/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin  Here is the output of which python: /usr/bin/python  Any tips for how to fix this? I'm on MacOS Big Sur 11.6.

Answer: Basically pip installs the packages under its related python directory, in a directory called site-packages (most likely, I'm not a python expert tbh). This is not included in the path you provided. First, ask pip to show the location to the package: pip show openai  The output would be something like this: Name: openai Version: 0.22.0 Summary: Python client library for the OpenAI API Home-page: https://github.com/openai/openai-python Author: OpenAI Author-email: support@openai.com License: Location: /Users/<USER>/DIR/TO/SOME/PYTHON/site-packages Requires: numpy, openpyxl, pandas, pandas-stubs, requests, tqdm Required-by:  So your package will be available in /Users/<USER>/DIR/TO/SOME/PYTHON/site-packages/openai  Either add /Users/<USER>/DIR/TO/SOME/PYTHON/site-packages/ to your path, or use the complete address to your package, or try to access it using your python: python -m openai # -m stands for module  To get more information about the -m flag, run python --help. Update So as you mentioned in the comments, you get permission denied after you add the directory to your package. This actually means that the package exists, but it's not permitted by your OS to execute. This is the thing you have to do, locate your package, and then: sudo chmod +x /PATH/TO/script  And the reason you're getting command not found after you use sudo directly with the package, is that you update your path variable in zsh, but when you use sudo, superuser uses sh instead of zsh.
Question: In order to download OpenCV on through Anaconda prompt, I run the following: conda install -c conda-forge opencv However, whenever I try to download, there I get the messages of failed with initial frozen solve. Retrying with flexible solve. Failed with repodata from current_repodata.json, will retry with next repodata source This would continue as the prompt tries to then diagnose what conflicts there are in my system that could prevent OpenCV from downloading. I kept my laptop on over night, but when I woke up in the morning, there was still diagnosing for potential conflicts going on. I'm not too sure what to do at this point. I just started trying again, but the same issues are being experienced. I am trying to download OpenCV so that I can import cv2 to work on machine learning projects for object/image detection. I have also tried pip install -c anaconda opencv but am having the same issues.

Answer: Please note that to import cv2, the library/package to install is called opencv-python. From Jupyter notebook, you can try !pip install opencv-python If you're using anaconda, you can try conda install -c conda-forge opencv-python
Question: I can do it like this but there has to be better way: arr = [] if len(arr) > 0: first_or_None = arr[0] else: first_or_None = None  If I just do arr[0] I get IndexError. Is there something where I can give default argument?

Answer: I think the example you give is absolutely fine - it is very readable and would not suffer performance issues. You could use the ternary operator python equivalent, if you really want it to be shorter code: last_or_none = arr[0] if len(arr) > 0 else None
Question: I'm building docs for a python project using Sphinx and readthedocs. For some reason, the code blocks aren't rendering after building the docs, and inline code (marked with backticks) appears italic. I've checked the raw build, there was a warning that the exomagpy module couldn't be found which I resolved by changing ".." to "../" in the os.path.abspath() part of conf.py, this didn't appear to change anything in the docs. There was a similar question here on stackoverflow, I tried the solution but it didn't change. The raw build can be found here: https://readthedocs.org/api/v2/build/17574729.txt The docs are being built from the "Develop" branch of the github page (link below) Here is the link to the github page: https://github.com/quasoph/exomagpy/tree/Develop The repo is structured like so: >.vscode >build/lib/exomagpy >docs >conf.py >rst files (.rst) >makefile >make.bat >exomagpy.egg-info >exomagpy >__init__.py >module files (.py) >.readthedocs.yml >requirements.txt >setup.py  Here are my files: conf.py import os import sys sys.path.insert(0, os.path.abspath("../")) # project info project = "exomagpy" root_doc = "index" release = "1.3.0" # general config extensions = ["sphinx.ext.autodoc","sphinx.ext.napoleon","sphinx.ext.autosummary" ] templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # html options html_theme = 'sphinx_rtd_theme' html_static_path = []  .readthedocs.yml version: 1 build: os: ubuntu-20.04 tools: python: "3.8" sphinx: configuration: docs/conf.py python: version: 3.8 install: - method: setuptools path: . - requirements: requirements.txt dependencies: - python=3.8  requirements.txt numpy matplotlib pandas tensorflow lightkurve requests sphinx==5.1.1 sphinx_rtd_theme==1.0.0  setup.py from setuptools import setup, find_packages setup( name = "exomagpy", version = "1.3.0", author = "Sophie", author_email = "email", url = "link", description = "description", packages = find_packages() )  Any help really appreciated, please let me know if more information is needed.

Answer: As @mzjn pointed out, a blank line is required between the code-block directive and the code that is supposed to be highlighted. https://raw.githubusercontent.com/quasoph/exomagpy/main/docs/tutorials.rst .. code-block:: python import exomagpy.predictExo exomagpy.predictExo.tess() exomagpy.predictExo.kepler()  Additionally for inline code literals, single backticks in reStructuredText are rendered as italics, as you realized, whereas in Markdown and its flavors it would render as an inline code literal. For reStructuredText, surround the literal with double backticks: ``inline_literal``
Question: Is there a way to have pandas.get_dummies output the numerical representation in one column rather than a separate column for each option? Concretely, currently when using pandas.get_dummies it gives me a column for every option:     Size Size_Big Size_Medium Size_Small     Big 1 0 0   Medium 0 1 0   Small 0 0 1     But I'm looking for more of the following output:     Size Size_Numerical     Big 1   Medium 2   Small 3    

Answer: You don't want dummies, you want factors/categories. Use pandas.factorize: df['Size_Numerical'] = pd.factorize(df['Size'])[0] + 1  output:  Size Size_Numerical 0 Big 1 1 Medium 2 2 Small 3
Question: There are many threads regarding Cannot find implementation or library stub for module named... error, but there's no associated error code. I'd like to disable this completely. How might I go about doing that?

Answer: To disable the warning: Pass the --ignore-missing-imports flag on the CLI. Or if using a config file: For mypy.ini or setup.cfg [mypy] ignore_missing_imports = true  For pyproject.toml [tool.mypy] ignore_missing_imports = true  This will disable warning for all modules. You can set it on a per module basis, but you'll have to check the docs for examples on how to do that.
Question: I have a dataframe of the following form:     Year 1 Grade Year 2 Grade Year 3 Grade Year 4 Grade Year 1 Students Year 2 Students Year 3 Students Year 4 Students     60 70 80 100 20 32 18 25     I would like to somehow transpose this table to the following format:     Year Grade Students     1 60 20   2 70 32   3 80 18   4 100 25     I created a list of years and initiated a new dataframe with the "year" column. I was thinking of matching the year integer to the column name containing it in the original DF, match and assign the correct value, but got stuck there.

Answer: Here's one way to do it. Feel free to ask questions about how it works. import pandas as pd cols = ["Year 1 Grade", "Year 2 Grade", "Year 3 Grade" , "Year 4 Grade", "Year 1 Students", "Year 2 Students", "Year 3 Students", "Year 4 Students"] vals = [60,70,80,100,20,32,18,25] vals = [[v] for v in vals] df = pd.DataFrame({k:v for k,v in zip(cols,vals)}) grades = df.filter(like="Grade").T.reset_index(drop=True).rename(columns={0:"Grades"}) students = df.filter(like="Student").T.reset_index(drop=True).rename(columns={0:"Students"}) pd.concat([grades,students], axis=1)
Question: I would like to ignore a specific line in static code analysis. For Flake8, I'd use the syntax # noqa: F401. For pylint, I'd use the syntax # pylint: disable=unused-import. As I am working on a code generation framework, I would like the code to support both linters. Is there a way to combine both directives such that both of them are correctly detected?

Answer: both of these combinations work for me: import os # noqa: F401 # pylint:disable=unused-import import sys # pylint:disable=unused-import # noqa: F401
Question: I'm unable to import from Python file in another directory. Directory structure: some_root/ - __init__.py - dir_0/ - __init__.py - dir_1/ - __init__.py - file_1.py - dir_2/ - __init__.py - file_2.py  file_1.py has some exported member: # file_1.py def foo(): pass  file_2.py tries to import member from file_1.py: # file_2.py from dir_0.dir_1.file_1 import foo  But not absolute nor relative import seems work. How to do Python's imports correctly? If I could avoid using sys.path.insert it would be nice, but if there's no way around this then I guess that's how things stand.

Answer: Don't mess with the search path You are right not messing around with sys.path. This is not recommended and always just an ugly workaround. There are better solutions for this. Restructure your folder layout See official Python docs about packaging. Distinguish between project folder and package folder. We assume your some_root folder as the package folder which is also the name of the package (used in import statements). And it is recommended to put the package folder into a folder named src. Above it is the project folder some_project. This project folder layout is also known as the "Src-Layout". In your case it should look like this. some_project └── src └── some_root ├── dir_0 │ ├── dir_1 │ │ ├── file_1.py │ │ └── __init__.py │ ├── dir_2 │ │ ├── file_2.py │ │ └── __init__.py │ └── __init__.py └── __init__.py  Make your package installable Create a some_project/setup.cfg with that content. Keep the line breaks and indention in line 5 and 6. They have to be like this but I don't know why. [metadata] name = some_project [options] package_dir= =src packages = find: zip_safe = False python_requires = >= 3 [options.packages.find] where = src exclude = tests* .gitignore  Create some_project/setup.py with that content: from setuptools import setup setup()  "Install" the package This is not a usual installation. Please see Developement Mode to understand what this really means. The package is not copied into /usr/lib/python/site-packages; only links are created. Navigate into the project folder some_project and run python3 -m pip install --editable .  Don't forget the . at the end. Depending on your OS and environment maybe you have to replace python3 with py -3 or python or something else. Import Your file_2.py import some_root import some_root.dir_0 import some_root.dir_0.dir_1 from some_root.dir_0.dir_1 import file_1 file_1.foo()  But as others said in the comments. Improve your structure of files and folders and reduce its complexity.
Question: I'm trying to install Odoo 15.0 on mac (python 3.7) when i come to run the command: pip3 install -r requirements.txt  I got this error message: Traceback (most recent call last): File "/usr/local/opt/python@3.7/bin/pip3", line 10, in <module> from importlib.metadata import distribution ModuleNotFoundError: No module named 'importlib.metadata' 

Answer: Try installing this lib manually, using : pip install importlib-metadata or pip3 install importlib-metadata
Question: Suppose there are two lists: l1 = [2,2,3] l2 = ['a','b','c']  I wonder how one finds the product of the two such that the output would be: #output: ['a','a','b','b','c','c','c']  if I do: l3 = [] for i in l2: for j in l1: l3.append(i)  I get: ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c']  which is wrong, I wonder where am I making the mistake?

Answer: The loop for j in l1: will iterate 3 times every time (because you have 3 items in list l1). Try: out = [b for a, b in zip(l1, l2) for _ in range(a)] print(out)  Prints: ['a', 'a', 'b', 'b', 'c', 'c', 'c']