eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How does Poetry associate a project to its virtual environment in ~/.cache/pypoetry/virtualenvs? I can't find any link inside the project, e.g. grep -ie NKJBdMnE . returns nothing. poetry env info: Virtualenv Python: 3.10.11 Implementation: CPython Path: /home/lddpro/.cache/pypoetry/virtualenvs/lxxo-NKJBdMnE-py3.10 Executable: /home/lddpro/.cache/pypoetry/virtualenvs/lxxo-NKJBdMnE-py3.10/bin/python Valid: True System Platform: linux OS: posix Python: 3.10.11 Path: /var/lang Executable: /var/lang/bin/python3.10  poetry config --list: [lddpro@0a0aecf400ca lddpro-bff]$ poetry config --list cache-dir = "/home/lddpro/.cache/pypoetry" experimental.new-installer = true experimental.system-git-client = false installer.max-workers = null installer.no-binary = null installer.parallel = true virtualenvs.create = true virtualenvs.in-project = null virtualenvs.options.always-copy = false virtualenvs.options.no-pip = false virtualenvs.options.no-setuptools = false virtualenvs.options.system-site-packages = false virtualenvs.path = "{cache-dir}/virtualenvs" # /home/lddpro/.cache/pypoetry/virtualenvs virtualenvs.prefer-active-python = false virtualenvs.prompt = "{project_name}-py{python_version}"  https://python-poetry.org/docs/configuration/

Answer: The path to the virtual environment is generated on the fly. It is not stored anywhere. The 8 letters NKJBdMnE are a part of the SHA256 of the working directory of you project. If you move your project into a different directory, poetry will use a different virtual environment as the SHA256 will be different. You can see the exact algorithm used in the poetry source code: env.py#L1196-L1204
Question: I want to convert file1.hic file into .cool format using hic2cool, which is written in Python. I converted the files using command line: hic2cool convert file1.hic file1.cool -r 10000  Traceback: Traceback (most recent call last): File "/home/melchua/.local/bin/hic2cool", line 5, in <module> from hic2cool.__main__ import main File "/home/melchua/.local/lib/python3.9/site-packages/hic2cool/__init__.py", line 2, in <module> from .hic2cool_utils import ( File "/home/melchua/.local/lib/python3.9/site-packages/hic2cool/hic2cool_utils.py", line 27, in <module> import cooler File "/home/melchua/.local/lib/python3.9/site-packages/cooler/__init__.py", line 14, in <module> from .api import Cooler, annotate File "/home/melchua/.local/lib/python3.9/site-packages/cooler/api.py", line 12, in <module> from .core import (get, region_to_offset, region_to_extent, RangeSelector1D, File "/home/melchua/.local/lib/python3.9/site-packages/cooler/core.py", line 3, in <module> from pandas.api.types import is_categorical ImportError: cannot import name 'is_categorical' from 'pandas.api.types' (/home/melchua/.local/lib/python3.9/site-packages/pandas/api/types/__init__.py) 

Answer: The latest version of pandas 2.0.1 seems to not have is_categorical but instead it has is_categorical_dtype. Seems that in hic2cool, pandas version is not pinned to the one that has that. I suggest installing a previous version of pandas before the changes took place. Or install a newer version of cooler as they updated their library to use is_categorical_dtype
Question: so currently i have this python bot that when a user tries to store something the author name along with the discriminator('#') is stored as well return "**Hello** **" + message.author.name + "#" + message.author.discriminator + "** ** :wave: " so my question is when they remove the discriminator and apply the new username system, how can i change that in my bot that it stores the username? also am i going to need to upgrade my discord python package and will they add something like this? message.author.username because when i currently try to call message.author.username i get this 'User' object has no attribute 'username` 

Answer: The Discord.py package will be updated sometime after discord applies the username system. For now, you can't use the new system as the developers of Discord.py need to see how the new system works before they implement changes. As for your question: You will have to update the Discord.py package when an update eventually comes out, otherwise you will not be able to use the updated methods. We also don't know how storing the username will be executed, but message.author.username seems like a good guess of what the method will be like
Question: I am trying to convert this udf into this pandas udf, in order to avoid creating two pandas udfs. Convert this: @udf("string") def splitEmailUDF(email: str, position: int) -> str: return email.split("@")[position]  into this in one pandas udf --- position ??? Datatype or something else! from pyspark.sql.functions import pandas_udf, PandasUDFType @pandas_udf("string") def splitEmailUDFVec(email: pd.Series, position: ???????) -> pd.Series: return email.str.split("@").str[position]  Of course I can always create two pandas_udfs: from pyspark.sql.functions import pandas_udf @pandas_udf("string") def splitFirstNameUDFVec(email: pd.Series) -> pd.Series: return email.str.split("@").str[0] @pandas_udf("string") def splitDomainUDFVec(email: pd.Series) -> pd.Series: return email.str.split("@").str[1]  Any help will be appreciated!

Answer: Setup df.show() +------------+ | email| +------------+ | foo@bar.com| |baz@spam.com| +------------+  Define a wrapper function which takes email and pos as arguments and returns the underlying pandas udf function def split(email, pos): @F.pandas_udf('string') def _split(email: pd.Series) -> pd.Series: return email.str.split('@').str[pos] return _split(email) df = df.withColumn('firstname', split('email', 1))  Result df.show() +------------+--------+ | email| domain| +------------+--------+ | foo@bar.com| bar.com| |baz@spam.com|spam.com| +------------+--------+   Alternatively a better/efficient approach is to use regex extraction if your goal is to only split the email address into its name and domain components. f = lambda n: F.regexp_extract('email', '(.*)@(.*)', n) df = df.select('*', f(1).alias('firstname'), f(2).alias('domain'))  Result df.show() +------------+---------+--------+ | email|firstname| domain| +------------+---------+--------+ | foo@bar.com| foo| bar.com| |baz@spam.com| baz|spam.com| +------------+---------+--------+
Question: I was trying to come up with a use case for the new @enum.nonmember decorator in Python 3.11. The docs clearly mention it is a decorator meant to be applied to members. However, when I tried literally decorating a member directly: import enum class MyClass(enum.Enum): A = 1 B = 2 @enum.nonmember C = 3  this results in an error as: Traceback (most recent call last): File "C:\Program Files\Python311\Lib\code.py", line 63, in runsource code = self.compile(source, filename, symbol) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\codeop.py", line 153, in __call__ return _maybe_compile(self.compiler, source, filename, symbol) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\codeop.py", line 73, in _maybe_compile return compiler(source, filename, symbol) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\codeop.py", line 118, in __call__ codeob = compile(source, filename, symbol, self.flags, True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<input>", line 9 C = 3 ^ SyntaxError: invalid syntax  However, if I had declared an atribute as a property or a descriptor it also wouldn't become an Enum member... So how, when and why do you use @enum.nonmember?

Answer: You would use it like so: import enum class MyClass(enum.Enum): A = 1 B = 2 C = enum.nonmember(3)  As far as I can tell, the only reason why it is called a decorator, is because of nested classes. Currently, class MyClass(enum.Enum): A = 1 B = 2 class MyNestedClass: pass  makes MyClass.MyNestedClass into one of the members of MyClass. This will change in 3.13. So if you want the new behaviour now, you can use: class MyClass(enum.Enum): A = 1 B = 2 @enum.nonmember class MyNestedClass: pass  In 3.13, if you want the current behaviour of making nested classes members, you can use class MyClass(enum.Enum): A = 1 B = 2 @enum.member class MyNestedClass: pass  There is no reason to use enum.nonmember as a decorator on a method, since methods are already excluded from being members, but I think you could use enum.member on one to be able to define a method member if you wanted. Not sure why you would, though.
Question: I'm trying to make a REST api (begginer) but when i tried to initialize the server from this code: from flask import Flask app = Flask(__name__) if __name__=='__main__': app.run(debug=True, port=4000)   i get this error in the prompt:  from watchdog.events import EVENT_TYPE_OPENED ImportError: cannot import name 'EVENT_TYPE_OPENED' from 'watchdog.events' (C:\ ********* \Python\Python310\lib\site-packages\watchdog\events.py)  I'm expecting something like this (Min 8:27): https://www.youtube.com/watch?v=GMppyAPbLYk&ab_channel=TechWithTim

Answer: Try to do the following: pip install --upgrade watchdog
Question: Suppose I have the following dataarray containing observations for different locations over time: import numpy as np import pandas as pd import xarray as xr np.random.seed(42) data = xr.DataArray( np.random.randint(1,100, (36, 3)), dims=("time", "location"), coords={ "time": pd.date_range("2022-01-01", periods=36, freq="10D"), "location": ["A", "B", "C"] }, name="observations" )  and now I calculate the monthly average and combine it with the observations to a dataset: monthly_avg = data.groupby("time.month").mean() data = data.to_dataset() data["average"] = monthly_avg  giving me  How can is set the indices correctly (if possible) so when I run: data.sel(time="2022-01-01")  I get a subset of the dataset for one time, all locations and one monthly average (which corresponds to the selected time)? At the moment when I run this I get  returning all monthly averages for the timestep. Conversely, when I run data.sel(month=1)  I'd like a subset with only the timesteps that are in January.

Answer: To get the selection return what you want, I would first compute the monthly averages and repeat them to match the original time-dimension. Then I would create a multi-index, such that you can select either the specific date or the month. #setup test data import numpy as np import pandas as pd import xarray as xr np.random.seed(42) data = xr.DataArray( np.random.randint(1,100, (36, 3)), dims=("time", "location"), coords={ "time": pd.date_range("2022-01-01", periods=36, freq="10D"), "location": ["A", "B", "C"] }, name="observations" ) #compute monthly array and repeat with list comprehension data=data.to_dataset() monthly_avg = data.groupby("time.month").mean()['observations'].values data['average']=(('time','location'),np.array([monthly_avg[i-1,:] for i in data.time.dt.month])) #add month and create multiindex data['month']=data.time.dt.month data=data.set_index(day_month=['time','month'])  You can then run the selection to get what you want. print(data.sel(time="2022-01-01")) <xarray.Dataset> Dimensions: (location: 3, month: 1) Coordinates: * location (location) <U1 'A' 'B' 'C' * month (month) int64 1 time <U10 '2022-01-01' Data variables: observations (month, location) int64 52 93 15 average (month, location) float64 70.5 82.25 33.75  print(data.sel(month=1)) <xarray.Dataset> Dimensions: (location: 3, time: 4) Coordinates: * location (location) <U1 'A' 'B' 'C' * time (time) datetime64[ns] 2022-01-01 2022-01-11 ... 2022-01-31 month int64 1 Data variables: observations (time, location) int64 52 93 15 72 61 21 83 87 75 75 88 24 average (time, location) float64 70.5 82.25 33.75 ... 70.5 82.25 33.7  This gives repeated values for the second command. Maybe there is a better way to set up the multi-index. You can have a look at the pandas multiindex documentation: https://pandas.pydata.org/docs/user_guide/advanced.html: or into stack/unstack in xarray: https://xarray.pydata.org/en/v0.7.2/reshaping.html#stack-and-unstack in case you haven't done so before.
Question: The code below just runs fine with Python 3.8.10 but does not run in Python 3.10. Any idea what could be the problem? import pandas as pd import requests url = "https://coinmarketcap.com/new/" page = requests.get(url,headers={'User-Agent': 'Mozilla/5.0'}, timeout=1) pagedata = page.text usecols = ["Name", "Symbol", "1h", "24h", "MarketCap"] df = pd.read_html(page.text)[0] df[["Name", "Symbol"]] = df["Name"].str.split(r"\d+", expand=True) df = (df.rename(columns={"Fully Diluted Market Cap": "MarketCap"})[usecols] .sort_values("24h", ascending=False, key=lambda ser: ser.str.replace("%", "").astype(float)) .replace(r"^\$", "", regex=True) ) numcols = df.columns[~df.columns.isin(['Name'])] df = df.head(5).to_markdown(index=True) print (df)  Current Output: Traceback (most recent call last): df[["Name", "Symbol"]] = df["Name"].str.split(r"\d+", expand=True) .... .... ValueError: Columns must be same length as key  Correct Output: (Output in Python 3.8) | | Name | Symbol | 1h | 24h | MarketCap | |---:|:------------|:----------|:-------|:---------|:------------| | 3 | Shrekt |4HREK | 23.82% | 2536.51% | 342,357 | | 8 | BLAZE |TOKEN9BLZE | 1.07% | 106.71% | 3,828,088 | | 26 | Goner27 |GONER | 6.32% | 88.09% | 1,094,010 | | 14 | Party Hat15 |PHAT | 13.34% | 81.64% | 60,136 | | 29 | PepeChat |30PPC | 48.01% | 78.25% | 431,159 | 

Answer: I think it has to do with one of the values (NOOT (BRC-20)4NOOT) found in the column Name. To handle this, we can try to split on the last number found in each row of this column. Replace this : df[["Name", "Symbol"]] = df["Name"].str.split(r"\d+", expand=True)  By this : df[["Name", "Symbol"]] = df["Name"].str.split(r"\d+(?!.*\d)", expand=True)   Regex [demo]  Output : print(df) | | Name | Symbol | 1h | 24h | MarketCap | |---:|:------------|:---------|:-------|:---------|:------------| | 5 | Shrekt | HREK | 54.61% | 1124.57% | 159,013 | | 10 | BLAZE TOKEN | BLZE | 2.40% | 109.53% | 3,880,242 | | 8 | CMC DOGE | CMCDOGE | 12.93% | 102.76% | 169,492 | | 28 | Goner | GONER | 1.37% | 88.66% | 1,050,089 | | 4 | nomeme | NOMEME | 53.86% | 86.14% | 4,603,393 |
Question: I have table similar to this one: index name_1 path1 name_2 path2 0 Roy path/to/Roy Anne path/to/Anne 1 Anne path/to/Anne Roy path/to/Roy 2 Hari path/to/Hari Wili path/to/Wili 3 Wili path/to/Wili Hari path/to/Hari 4 Miko path/to/miko Lin path/to/lin 5 Miko path/to/miko Dan path/to/dan 6 Lin path/to/lin Miko path/to/miko 7 Lin path/to/lin Dan path/to/dan 8 Dan path/to/dan Miko path/to/miko 9 Dan path/to/dan Lin path/to/lin ...  As you can see, the table kind of showing relationship between entities - Roi is with Anne, Wili with Hari, Lin with Dan and with Miko. The table is actually showing overlap data , meaning, Hari and wili for example, have the same document, and I would like to remove one of them not to have duplicated files. In order to do this, I would like to create new table that has only one name in relationship, so I can later create list of paths to remove. The result table will look like this : index name_1 path1 name_2 path2 0 Roy path/to/Roy Anne path/to/Anne 1 Hari path/to/Hari Wili path/to/Wili 2 Miko path/to/miko Lin path/to/lin 3 Miko path/to/miko Dan path/to/dan  The idea is that I'll use the values of "path2" to remove files with this path, and will still have the files in path1. for that reason, this line: 4 Lin path/to/lin Dan path/to/dan  is missing, as it will be removed using Miko... any ideas how to do this ? :) Edit: I have tried this based on this answer: df_2= df[~pd.DataFrame(np.sort(df.values,axis=1)).duplicated()]  And it's true that I get less rows in my dataframe (it has 695 and I got now 402) , but, I still have the first lines like this: index name_1 path1 name_2 path2 0 Roy path/to/Roy Anne path/to/Anne 1 Anne path/to/Anne Roy path/to/Roy ...  meaning I still get the same issue

Answer: You can use frozenset to detect duplicates: out = (df[~df[['name_1', 'name_2']].agg(frozenset, axis=1).duplicated()] .loc[lambda x: ~x['path2'].isin(x['path1'])]) # OR out = (df[~pd.DataFrame(np.sort(df.values,axis=1)).duplicated()] .query('~path1.isin(path2)'))  Output: >>> out name_1 path1 name_2 path2 0 Roy path/to/Roy Anne path/to/Anne 2 Hari path/to/Hari Wili path/to/Wili 5 Miko path/to/miko Dan path/to/dan 7 Lin path/to/lin Dan path/to/dan
Question: I have an HTML file which contains XML at the bottom of it and enclosed by comments, it looks like this: <!DOCTYPE html> <html> <head> *** </head> <body> <div class="panel panel-primary call__report-modal-panel"> <div class="panel-heading text-center custom-panel-heading"> <h2>Report</h2> </div> <div class="panel-body"> <div class="panel panel-default"> <div class="panel-heading"> <div class="panel-title">Info</div> </div> <div class="panel-body"> <table class="table table-bordered table-page-break-auto table-layout-fixed"> <tr> <td class="col-sm-4">ID</td> <td class="col-sm-8">1</td> </tr> </table> </div> </div> </body> </html> <!--<?xml version = "1.0" encoding="Windows-1252" standalone="yes"?> <ROOTTAG> <mytag> <headername>BASE</headername> <fieldname>NAME</fieldname> <val><![CDATA[Testcase]]></val> </mytag> <mytag> <headername>BASE</headername> <fieldname>AGE</fieldname> <val><![CDATA[5]]></val> </mytag> </ROOTTAG> -->  Requirement is to parse the XML which is in comments in above HTML. So far I have tried to read the HTML file and pass it to a string and did following: with open('my_html.html', 'rb') as file: d = str(file.read()) d2 = d[d.index('<!--') + 4:d.index('-->')] d3 = "'''"+d2+"'''"  this is returning XML piece of data in string d3 with 3 single qoutes. Then trying to read it via Etree: ET.fromstring(d3)  but it is failing with following error: xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 2 need some help to basically:  Read HTML take out snippet with XML piece which is commented at bottom of HTML take that string and pass to ET.fromString() function, but since this function takes string with triple qoutes, it is not formatting it properly and hence throwing the error 

Answer: You already have been on the right path. I put your HTML in the file and it works fine like following. import xml.etree.ElementTree as ET with open('extract_xml.html') as handle: content = handle.read() xml = content[content.index('<!--')+4: content.index('-->')] document = ET.fromstring(xml) for element in document.findall("./mytag"): for child in element: print(child, child.text)
Question: Can PEP compliant type hints be automatically added to generated source code, or dynamically created, for python and gRPC? Specifically in the basics tutorial in the client section for feature = stub.GetFeature(point) I would like my IDE to know and check that point is type Point in the *_pb2.py and feature is type Feature with an attribute location: Point. Thank you.

Answer: Type hints aren't yet (!?) available for Python gRPC, see Issue 29041 You can generate type hints for the Protobuf messages only using the --pyi_out=${PWD} flag when running protoc directly or indirectly with python3 -m grpc.tools.protoc
Question: I have a file multiply.py with the following contents: from typing import NamedTuple class Result(NamedTuple): product: int desc: str def multiply(a: int, b: int) -> Result: return Result( product=a * b, desc=f"muliplied {a} and {b}", ) x=4 y=5 print(multiply(x, y))  If I run it just like that it of course yields the expected result: $ python multiply.py Result(product=20, desc='muliplied 4 and 5')  However I'm trying to run it with exec function from main.py: from pathlib import Path gl, lo = {}, {} exec(Path("multiply.py").read_text(), gl, lo)  and this time the output is disappointing: $ python main.py Traceback (most recent call last): File "main.py", line 4, in <module> exec(Path("multiply.py").read_text(), gl, lo) File "<string>", line 16, in <module> File "<string>", line 9, in multiply NameError: name 'Result' is not defined  Why is that? Can't I create new types in the code executed by exec?

Answer: Since you are passing two different dictionaries for locals and globals, from the docs:  If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition  Class bodies are not an enclosing scope, functions defined in the class body do not have access to variables defined in the class body. This is the reason that in a method definition you must use self. to refer to another method. To elaborate a little on what I mean about self, here's an example of actually just running the code you have in a class body, and seeing how it behaves: In [1]: class NS: ...: from typing import NamedTuple ...: ...: class Result(NamedTuple): ...: product: int ...: desc: str ...: ...: def multiply(a: int, b: int) -> Result: ...: return Result( ...: product=a * b, ...: desc=f"muliplied {a} and {b}", ...: ) ...: ...: In [2]: NS.multiply(1, 2) --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In [2], line 1 ----> 1 NS.multiply(1, 2) Cell In [1], line 9, in NS.multiply(a, b) 8 def multiply(a: int, b: int) -> Result: ----> 9 return Result( 10 product=a * b, 11 desc=f"muliplied {a} and {b}", 12 ) NameError: name 'Result' is not defined  In any case, exec is a function that is typically misused. There are times when it is perfectly fine, but often it is the case that when people reach for exec/eval there is a better way to do it in Python.
Question: I'd like to add hidden methods to pyo3 class methods implementation which will be invisible for Python. Example: #[pyclass] pub struct SomeItem; #[pymethods] impl SomeItem { #[new] pub fn new() -> Self { // visible for python constructor SomeItem; } pub fn method(&self) -> u8 { // visible for python method self.hidden_method() } #[pyo3(ignore)] // just for example fn hidden_method(&self) -> u8 { // invisible for python method (that must be able to return non-python type) 0 } } 

Answer: What about using a separate impl block? #[pymethods] impl SomeItem { #[new] pub fn new() -> Self { // visible for python constructor SomeItem; } pub fn method(&self) -> u8 { // visible for python method self.hidden_method() } } impl SomeItem { fn hidden_method(&self) -> u8 { 0 } }
Question: I'm working with the arrow dtypes with pandas, and my dataframe has a variable that should be categorical, but I can't figure out how to transform it into pyarrow data type for categorical data (dictionary) According to pandas (https://arrow.apache.org/docs/python/pandas.html#pandas-arrow-conversion), the arrow data type I should be using is dictionary. Usually, if you want pandas to use a pyarrow dtype you just add[pyarrow] to the name of the pyarrow type, for example dtype='string[pyarrow]'. I tried using dtype='dictionary[pyarrow]', but that yields the error:  data type 'dictionary[pyarrow]' not understood  I also tried 'categorical[pyarrow]', or 'category[pyarrow]', pyarrow.dictionary, pyarrow.dictionary(pyarrow.int16(),pyarrow.string()), and they didn't work either. How can i use dictionary dtype on a pandas series? pd.Series(['Chocolate','Candy','Waffles'], dtype='what_to_put_here????')

Answer: I believe pd.ArrowDtype is required: dtype=pd.ArrowDtype(pa.dictionary(pa.int16(), pa.string()))