eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working on a large Pandas DataFrame which needs to be converted into dictionaries before being processed by another API. The required dictionaries can be generated by calling the .to_dict(orient='records') method. As stated in the docs, the returned value depends on the orient option:  Returns: dict, list or collections.abc.Mapping Return a collections.abc.Mapping object representing the DataFrame. The resulting transformation depends on the orient parameter.  For my case, passing orient='records', a list of dictionaries is returned. When dealing with lists, the complete memory required to store the list items, is reserved/allocated. As my dataframe can get rather large, this might lead to memory issues especially as the code might be executed on lower spec target systems. I could certainly circumvent this issue by processing the dataframe chunk-wise and generate the list of dictionaries for each chunk which is then passed to the API. Furthermore, calling iter(df.to_dict(orient='records')) would return the desired generator, but would not reduce the required memory footprint as the list is created intermediately. Is there a way to directly return a generator expression from df.to_dict(orient='records') instead of a list in order to reduce the memory footprint?

Answer: There is not a way to get a generator directly from to_dict(orient='records'). However, it is possible to modify the to_dict source code to be a generator instead of returning a list comprehension: from pandas.core.common import standardize_mapping from pandas.core.dtypes.cast import maybe_box_native def dataframe_records_gen(df_): columns = df_.columns.tolist() into_c = standardize_mapping(dict) for row in df_.itertuples(index=False, name=None): yield into_c( (k, maybe_box_native(v)) for k, v in dict(zip(columns, row)).items() )  Sample Code: import pandas as pd df = pd.DataFrame({ 'A': [1, 2], 'B': [3, 4] }) # Using Generator for row in dataframe_records_gen(df): print(row) # For Comparison with to_dict function print("to_dict", df.to_dict(orient='records'))  Output: {'A': 1, 'B': 3} {'A': 2, 'B': 4} to_dict [{'A': 1, 'B': 3}, {'A': 2, 'B': 4}]   For more natural syntax, it's also possible to register a custom accessor: import pandas as pd from pandas.core.common import standardize_mapping from pandas.core.dtypes.cast import maybe_box_native @pd.api.extensions.register_dataframe_accessor("gen") class GenAccessor: def __init__(self, pandas_obj): self._obj = pandas_obj def records(self): columns = self._obj.columns.tolist() into_c = standardize_mapping(dict) for row in self._obj.itertuples(index=False, name=None): yield into_c( (k, maybe_box_native(v)) for k, v in dict(zip(columns, row)).items() )  Which makes this generator accessible via the gen accessor in this case: df = pd.DataFrame({ 'A': [1, 2], 'B': [3, 4] }) # Using Generator through registered custom accessor for row in df.gen.records(): print(row) # For Comparison with to_dict function print("to_dict", df.to_dict(orient='records'))  Output: {'A': 1, 'B': 3} {'A': 2, 'B': 4} to_dict [{'A': 1, 'B': 3}, {'A': 2, 'B': 4}]
Question: When you have a setup.py file, you can get the name of the package via the command: C:\some\dir>python setup.py --name  And this would print the name of the package to the command line. In an attempt to adhere to best practice, I'm trying to migrate away from setup.py by putting everything in setup.cfg since everything that was previously in setup.py was static content. But our build pipeline depends on being able to call python setup.py --name. I'm looking to rewrite the pipeline in such a way that I don't need to create a setup.py file. Is there way to get the name of the package when you have a setup.cfg but not a setup.py file?

Answer: Maybe using the ConfigParser Python module ? python -c "from configparser import ConfigParser; cf = ConfigParser(); cf.read('setup.cfg'); print(cf['metadata']['name'])"
Question: I have following piece of code for github actions: name: Python application on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Install dependencies run: / python -m pip install --upgrade pip python -m pip install numpy pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - name: Test Selenium run: / python -m pytest -v -m devs  But when I commit, and actions start running, I get this error: Run / python -m pip install --upgrade pip python -m pip install numpy pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi / python -m pip install --upgrade pip python -m pip install numpy pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi shell: /usr/bin/bash -e {0} /home/runner/work/_temp/317511f5-0874-43d8-a0ae-2601804ff811.sh: line 1: syntax error near unexpected token `then' Error: Process completed with exit code 2.  And this is just under my Install dependencies. Am I missing something super obvious? Thank you in advance for the help.

Answer: Yes - you have a simple mistake there :) To have multiple commands under run you have to use: run: | not \ \ is used later on to have one bash command split into multiple lines name: Python application on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Install dependencies run: | python -m pip install --upgrade pip python -m pip install numpy pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - name: Test Selenium run: | python -m pytest -v -m devs
Question: Let's say we have a trivial function that calls open() but with a fixed argument: def open_for_writing(*args, **kwargs): kwargs['mode'] = 'w' return open(*args, **kwargs)  If I now try to call open_for_writing(some_fake_arg = 123), no type checker (e.g. mypy) can tell that this is an incorrect invocation: it's missing the required file argument, and is adding another argument that isn't part of the open signature. How can I tell the type checker that *args and **kwargs must be a subset of the open parameter spec? I realise Python 3.10 has the new ParamSpec type, but it doesn't seem to apply here because you can't get the ParamSpec of a concrete function like open.

Answer: I think out of the box this is not possible. However, you could write a decorator that takes the function that contains the arguments you want to get checked for (open in your case) as an input and returns the decorated function, i.e. open_for_writing in your case. This of course only works with python 3.10 or using typing_extensions as it makes use of ParamSpec from typing import TypeVar, ParamSpec, Callable, Optional T = TypeVar('T') P = ParamSpec('P') def take_annotation_from(this: Callable[P, Optional[T]]) -> Callable[[Callable], Callable[P, Optional[T]]]: def decorator(real_function: Callable) -> Callable[P, Optional[T]]: def new_function(*args: P.args, **kwargs: P.kwargs) -> Optional[T]: return real_function(*args, **kwargs) return new_function return decorator @take_annotation_from(open) def open_for_writing(*args, **kwargs): kwargs['mode'] = 'w' return open(*args, **kwargs) open_for_writing(some_fake_arg=123) open_for_writing(file='')  As shown here, mypy complains now about getting an unknown argument.
Question: In the below example, I am trying to install gradio package but I have seen -q flag is used in some tutorials to install packages.What does '-q' flag means in pip install -q ? pip install -q gradio

Answer: Running pip3 --help gives you this:  -q, --quiet Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).  So the -q option reduces the output produced by pip. It does not affect the installation process. It is a general option.
Question: I know that in general python only makes new scopes for classes, functions etc., but I'm confused by the as statement in a try/except block or context manager. Variables assigned inside the block are accessible outside it, which makes sense, but the variable bound with as itself is not. So this fails: try: raise RuntimeError() except RuntimeError as error: pass print(repr(error))  but this succeeds: try: raise RuntimeError() except RuntimeError as e: error = e print(repr(error))  What's going on with the variable bound with as, and why don't normal python scoping rules apply? The PEP indicates that it's just a normally bound python variable, but that doesn't seem to be the case.

Answer: As explained in PEP 3110, as well as current documentation, variables bound with as in an except block are explicitly and specially cleared at the end of the block, even though they share the same local scope. This improves the immediacy of garbage collection. The as syntax was originally not available for exceptions in 2.x; it was backported for 2.6, but the old semantics were preserved. The same does not apply to with blocks: >>> from contextlib import contextmanager >>> @contextmanager ... def test(): ... yield ... >>> with test() as a: ... pass ... >>> a # contains None; does not raise NameError >>> >>> def func(): # similarly within a function ... with test() as a: ... pass ... return a ... >>> func() >>>  The behaviour is specific to the except block, not to the as keyword.
Question: How can I escape a : inside of a string passed to text() to prevent SQLAlchemy from treating it like a bindparameter? conn.execute(text("select 'My favorite emoticon is :p' from dual")).fetchone()  Will result in: sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A value is required for bind parameter 'p' (Background on this error at: http://sqlalche.me/e/14/cd3x)  ' It's a bit confusing because from the context of selecting a string from the database select 'foo :bar baz' a bindparameter doesn't make much sense here. It looks like I can use a \ to escape this, but it says it is deprecated: >>> conn.execute(text("select 'My favorite emoticon is \:p' from dual")).fetchone() <stdin>:1: DeprecationWarning: invalid escape sequence \: ('My favorite emoticon is :p',) 

Answer: As mentioned in the docs:  For SQL statements where a colon is required verbatim, as within an inline string, use a backslash to escape  But remember that the backslash is also the escape character in Python string literals, so text("select 'My favorite emoticon is \:p' from dual")  is incorrect because Python will want to interpret \: as an escape character. We need to use either a "raw string" (r"") text(r"select 'My favorite emoticon is \:p' from dual")  or escape the backslash itself text("select 'My favorite emoticon is \\:p' from dual")
Question: I've read all of the other questions on this error and frustratingly enough, none give a solution that works. If I run pip install sentencepiece in the cmd line, it gives me the following output.  src/sentencepiece/sentencepiece_wrap.cxx(2809): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2 [end of output]  I'm running python 3.10.1 and pip 22.0.3 . *I have the following Microsoft Visual C++ programs on my windows machine,which I've just done a fresh install of as it was complaining of not having a particular C++ program. MS VC++  I've even added the .exe file to my PATH variables but still I get the same error. Am I missing a particular Microsoft program on my pc?

Answer: I haven't seen this problem in Windows, but for Linux, I would normally reinstall Python after installing the dependencies (such as the MSVC thing). In that case this is especially helpful because I'm often rebuilding (compiling and other related steps) Python/Pip. Could also just be an error specific to the module and Python version combo you're trying.  From a discussion in the comments: I have the pyenv-win version manager, so I was able to create venvs and test this for you. With Python 3.10.2, it fails; with Python 3.8.10, it's successful. So, yes, reinstalling does seem to be worthy of your time.
Question: I am a beginner to Selenium python. I have tried to invoke the Edge browser with an existing profile(Default) with the following code. But it is throwing the following exception as soon as the execution starts. Can someone please help me with this? Am I missing something? edge_options = webdriver.EdgeOptions() edge_options.add_argument("user-data-dir = C:/Users/XYZ/AppData/Local/Microsoft/Edge/User Data/Default") edge_browser = webdriver.Edge(executable_path = "C:/Users/XYZ/ABC/msedgedriver.exe",options = edge_options ) edge_browser.maximize_window()  WebDriverException: unknown error: MSEdge failed to start: crashed. (chrome not reachable) (The process started from msedge location C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe is no longer running, so MSEdgeDriver is assuming that MSEdge has crashed.) Note: Edge browser is getting invoked and works properly when I run the code without the following line edge_options.add_argument("user-data-dir = C:/Users/XYZ/AppData/Local/Microsoft/Edge/User Data/Default") 

Answer: I came across the issue before, that's because there're running Edge processes in the background. The solution is you can back up your User Data folder in the same path and use that folder in selenium:  Back up your User Data folder in the same path. Here for example, I back up the User Data folder as User Data1:   Use User Data1 in your code to specify using Default profile when run Edge with Selenium: from selenium import webdriver from selenium.webdriver.edge.service import Service edge_options = webdriver.EdgeOptions() #Here you set the path of the back up profile ending with User Data1 not the profile folder edge_options.add_argument("user-data-dir=C:\\Users\\XYZ\\AppData\\Local\\Microsoft\\Edge\\User Data1") ser = Service("C:\\Users\\XYZ\\ABC\\msedgedriver.exe") edge_browser = webdriver.Edge(options = edge_options, service=ser) edge_browser.maximize_window()
Question: To motivate the question: In recent NumPy versions there is a performance issue with numpy.clip and in the corresponding issue a suggested workaround is to use numpy.core.umath.clip. I couldn't really find anything about the purpose of the numpy.core.umath or more generally the numpy.core.* modules. pydoc shows documentation for them but they seem not to be included in the online documentation. Therefore I am a bit unsure if it is safe (in the sense of having a stable API) to use functions from these modules or if they should rather be seen as internals that may change between releases?

Answer: The documentation of numpy.core states:  Please note that this module is private. All functions and objects are available in the main numpy namespace - use that instead.  This submodule is apparently not meant to be used by end-users. This is also why there is almost no online documentation about it. The functions in this submodule are more likely to change in the future than the regular ones in provided in numpy.
Question: Is there a way to cache poetry install command in Gitlab CI (.gitlab-ci.yml)? For example, in node yarn there is a way to cache yarn install (https://classic.yarnpkg.com/lang/en/docs/install-ci/ Gitlab section) this makes stages a lot faster.

Answer: GitLab can only cache things in the working directory and Poetry stores packages elsewhere by default:  Directory where virtual environments will be created. Defaults to {cache-dir}/virtualenvs ({cache-dir}\virtualenvs on Windows).  On my machine, cache-dir is /home/chris/.cache/pypoetry. You can use the virtualenvs.in-project option to change this behaviour:  If set to true, the virtualenv wil be created and expected in a folder named .venv within the root directory of the project.  So, something like this should work in your gitlab-ci.yml: before_script: - poetry config virtualenvs.in-project true cache: paths: - .venv
Question: I'm trying to build a Ubuntu 18.04 Docker image running Python 3.7 for a machine learning project. When installing specific Python packages with pip from requirements.txt, I get the following error: Collecting sklearn==0.0 Downloading sklearn-0.0.tar.gz (1.1 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] ERROR: Can not execute `setup.py` since setuptools is not available in the build environment. [end of output]  Although here the error arises in the context of sklearn, the issue is not specific to one library; when I remove that libraries and try to rebuild the image, the error arises with other libraries. Here is my Dockerfile: FROM ubuntu:18.04 # install python RUN apt-get update && \ apt-get install --no-install-recommends -y \ python3.7 python3-pip python3.7-dev # copy requirements WORKDIR /opt/program COPY requirements.txt requirements.txt # install requirements RUN python3.7 -m pip install --upgrade pip && \ python3.7 -m pip install -r requirements.txt # set up program in image COPY . /opt/program  What I've tried:  installing python-devtools, both instead of and alongside, python3.7-dev before installing requirements with pip; installing setuptools in requirements.txt before affected libraries are installed.  In both cases the same error arose. Do you know how I can ensure setuptools is available in my environment when installing libraries like sklearn?

Answer: As mentioned in comment, install setuptools with pip before running pip install -r requirements.txt. It is different than putting setuptools higher in the requirements.txt because it forces the order while the requirements file collect all the packages and installs them after so you don't control the order.
Question: My problem is as follows: I have written a python code, and I need to run it on a web page.Basically I need that whatever is on the console should be displayed as it is. I have no experience in web development and similar libraries, and I need to get this done in a short time. Kindly tell how should I proceed? Note: I might be plotting some graphs also. It would be great if they could be displayed all at once(sequentially) on the website

Answer:  https://brython.info/ https://skulpt.org/ https://pyodide.org/en/stable/  There are multiple python implementations in the browser: some are WebAssembly (WASM) and some are JavaScript. Is it a good idea to run python on browser as a replacement for JavaScript in 2022? No it is not; learn JavaScript. No in-browser python implementation can match JavaScript and its performance as of today and most probably ever.
Question: df.filter(pl.col("MyDate") >= "2020-01-01")  does not work like it does in pandas. I found a workaround df.filter(pl.col("MyDate") >= pl.datetime(2020,1,1))  but this does not solve a problem if I need to use string variables.

Answer: You can turn the string into a date type e.g. with .str.to_date() Building on the example above: import polars as pl from datetime import datetime df = pl.DataFrame({ "dates": [datetime(2021, 1, 1), datetime(2021, 1, 2), datetime(2021, 1, 3)], "vals": range(3) }) df.filter(pl.col('dates') >= pl.lit(my_date_str).str.to_date())  shape: (2, 2) ┌─────────────────────┬──────┐ │ dates ┆ vals │ │ --- ┆ --- │ │ datetime[μs] ┆ i64 │ ╞═════════════════════╪══════╡ │ 2021-01-02 00:00:00 ┆ 1 │ │ 2021-01-03 00:00:00 ┆ 2 │ └─────────────────────┴──────┘
Question: I'm struggling with Python enums. I created an enum class containing various fields: class Animal(Enum): DOG = "doggy" CAT = "cute cat"  I know that I can access this enum with value i.e. by passing Animal("doggy") I will have Animal.DOG. However I would like to achieve the same but the other way around, let's say I have "DOG" and I use it to find "doggy". Example: str_to_animal_enum("DOG") = Animal.DOG  I thought that I could rewrite init function of Enum class to check whether string name that we put as a parameter has representants in Enum names, if yes then we have a match. Sth like this: def __init__(cls, name): for enum in all_enums: if enum.name == name: return enum  But this solution looks very heavy in computation. Let's say that I have 100 names to find and for each name we have to make N iterations in the worst case scenario where N is number of enums in enum class. I think there should be something better but I just don't know what. Do you have any ideas how it could be implemented? Thanks :)

Answer: As per the manual: >>> Animal.DOG.value 'doggy' >>> Animal.DOG.name 'DOG' >>> # For more programmatic access, with just the enum member name as a string: >>> Animal['DOG'].value 'doggy'
Question: I was using a jupyter notebook inside VSCode and used ?? on the object to look the source code. The output showed :   Output exceeds the size limit. Open the full output data in a text editor  But when I click on it it opens the output in another window but everything is illegible.  What's going on here? What are those strange characters like esc[031m? How can I get rid of them when viewing the full output data?

Answer: Those are ANSI escape codes- particularly ones for colouring. If ANSI color support in edit buffer #38834 gets implemented, then this problem will sort of "go away" by default (though I imagine it could lead to different kinds of confusion). The IPython configuration docs have a section on terminal colours:  InteractiveShell.colors sets the colour of tracebacks and object info (the output from e.g. zip?). It may also affect other things if the option below is set to 'legacy'. It has four case-insensitive values: 'nocolor', 'neutral', 'linux', 'lightbg'. The default is neutral, which should be legible on either dark or light terminal backgrounds. linux is optimised for dark backgrounds and lightbg for light ones.  See the rest of those docs for more info. Ideally, the VS Code extension for IPython would strip out those ANSI escape sequences when showing the full output data in a text editor, but for now, you may be able to work around that by manual configuration by setting InteractiveShell.colors to 'nocolor', or putting the following code cell at the start of your notebook: %colors nocolor (see related docs here). This problem has been brought up on the microsoft/vscode-jupyter GitHub repo at least twice:  Truncated stacktraces illegible in text editor #10467 (closed pointing to another issue ticket: Output truncated with no way to view all output in the notebook itself #7096)  Error traces opened in text editor are not readable #11279 (closed as a duplicate of #10467)
Question: When faced with a large dataset, I need to spend a day using GridSearchCV() to train an SVM with the best parameters. How can I save the best estimator so that I can use this trained estimator directly when I start my computer next time?

Answer: By default, GridSearchCV does not expose or store the best model instance it only returns the parameter set that led to the highest score. If you want the best predictor, you have to specify refit=True, or if you are using multiple metrics refit=name-of-your-decider-metric. This will run a final training step using the full dataset and the best parameters found. To find the optimal parameters, GridSearchCV obviously does not use the entire dataset for training, as they have to split out the hold-out validation set. Now, when you do that, you can get the model via the best_estimator_ attribute. Having this, you can pickel that model using joblib and reload it the next day to do your prediction. In a mix of pseudo and real code, that would read like: from sklearn.model_selection import GridSearchCV from sklearn.svm import SVC from joblib import dump, load svc = SVC() # Probably not what you are using, but just as an example gcv = GridSearchCV(svc, parameters, refit=True) gcv.fit(X, y) estimator = gcv.best_estimator_ dump(estimator, "your-model.joblib") # Somewhere else estimator = load("your-model.joblib")