eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is it possible to install huggingface models as a project dependency? Currently it is downloaded automatically by the SentenceTransformer library, but this means in a docker container it downloads every time it starts. This is the model I am trying to use: https://huggingface.co/sentence-transformers/all-mpnet-base-v2 I have tried specifying the url as a dependency in my pyproject.toml: all-mpnet-base-v2 = {git = "https://huggingface.co/sentence-transformers/all-mpnet-base-v2.git", branch = "main"}  The first error I got was that the name was incorrect and it should be called train-script, which I renamed the dependency to, but I'm not sure if this is correct. Now I have: train-script = {git = "https://huggingface.co/sentence-transformers/all-mpnet-base-v2.git", branch = "main"}  However, now I get the following error:  Package operations: 1 install, 0 updates, 0 removals • Installing train-script (0.0.0 bd44305) EnvCommandError Command ['/srv/.venv/bin/pip', 'install', '--no-deps', '-U', '/srv/.venv/src/train-script'] errored with the following return code 1, and output: ERROR: Directory '/srv/.venv/src/train-script' is not installable. Neither 'setup.py' nor 'pyproject.toml' found. [notice] A new release of pip available: 22.2.2 -> 22.3.1 [notice] To update, run: pip install --upgrade pip at /usr/local/lib/python3.10/site-packages/poetry/utils/env.py:1183 in _run 1179│ output = subprocess.check_output( 1180│ cmd, stderr=subprocess.STDOUT, **kwargs 1181│ ) 1182│ except CalledProcessError as e: → 1183│ raise EnvCommandError(e, input=input_) 1184│ 1185│ return decode(output) 1186│ 1187│ def execute(self, bin, *args, **kwargs):  Is this possible? If not, is there a recommended way to bake the model download into a docker container so it doesn't need to be downloaded each time?

Answer: I was not able to find a native way to do this with project dependency files, so I did this using a multi-stage docker file. First I clone the model locally, then copy it into the appropriate /root/.cache/torch/ folder. Here is an example: FROM python:3.10.3 as model-download-stage RUN apt update && apt install git-lfs -y RUN git lfs install RUN git clone https://huggingface.co/sentence-transformers/all-mpnet-base-v2 /tmp/model RUN rm -rf /tmp/model/.git FROM python:3.10.3 COPY --from=model-download-stage /tmp/model /root/.cache/torch/sentence_transformers/sentence-transformers_all-mpnet-base-v2
Question: When I was trying to import matplotlib, I wrote import matplotlib.pyplot as plt  in my code. and this error occured. Traceback (most recent call last): File "C:\aiProjects\opencv\test.py", line 2, in <module> import matplotlib.pyplot as plt File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\__init__.py", line 113, in <module> from . import _api, _version, cbook, _docstring, rcsetup File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\rcsetup.py", line 27, in <module> from matplotlib.colors import Colormap, is_color_like File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\colors.py", line 56, in <module> from matplotlib import _api, _cm, cbook, scale File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\scale.py", line 22, in <module> from matplotlib.ticker import ( File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\ticker.py", line 138, in <module> from matplotlib import transforms as mtransforms File "C:\Users\blackhao\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\transforms.py", line 49, in <module> from matplotlib._path import ( ImportError: DLL load failed while importing _path: The specified module could not be found. Process finished with exit code 1  I tried several ways such as reinstalling matplotlib and pasting new "Msvcp71.dll" and "Msvcr71.dll" files in my system folders. Anyways to solve this problem?

Answer: ... reinstall the matplotlib python package with this argument --ignore-installed: pip3 install matplotlib --user --ignore-installed
Question: I'm trying to load tokenizer and seq2seq model from pretrained models. from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("ozcangundes/mt5-small-turkish-summarization") model = AutoModelForSeq2SeqLM.from_pretrained("ozcangundes/mt5-small-turkish-summarization")  But I got this error. File ~/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py:1028, in FileDescriptor.__new__(cls, name, package, options, serialized_options, serialized_pb, dependencies, public_dependencies, syntax, pool, create_key) 1026 raise RuntimeError('Please link in cpp generated lib for %s' % (name)) 1027 elif serialized_pb: -> 1028 return _message.default_pool.AddSerializedFile(serialized_pb) 1029 else: 1030 return super(FileDescriptor, cls).__new__(cls) TypeError: Couldn't build proto file into descriptor pool: duplicate file name (sentencepiece_model.proto)  I tried updating or downgrading the protobuf version. But I couldn't fix

Answer: I ran into the same issue when trying to use the microsoft/deberta-v3-small model. That is, at first it complained about not being able to find protobuf, and when I installed the latest, it asked for version 3.20.x. The issue happened after I downgraded to the lower version. Anyway, I was experimenting with it on a locally-running Jupyter notebook. Rerunning that cell didn't help. But when I chose to "Restart & Run All," the problem went away. Therefore, to solve your issue, I believe that you need to restart the Python instance where it cached the latest version of protobuf in the first place. Steps to reproduce:   from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("some_model") # outcome: some error asking for `protobuf`    pip install protobuf  Rerun the code above; some error asking for protobuf@3.20.x   pip uninstall protobuf pip install protobuf==3.20  Rerun the code above; same error as in OP  Restart the Python instance (notebook, app, etc.); ✓ no error
Question: I have a test that looks as following: @pytest.mark.parametrize('param', ['my param', 'my param 2']) def test_param(self,param): ...  This works fine when calling this test with python3 -m pytest -s -k "test_param"  However, if I want to target a specific test as following: python3 -m pytest -s -k "test_param[my param]"  I get the error message ERROR: Wrong expression passed to '-k': my param: at column 4: expected end of input; got identifier  Also, when my input string contains a quotation mark ', I get the error ERROR: Wrong expression passed to '-k': ... : at column 51: expected end of input; got left parenthesis  and if my string contains both " and ', I am completely unable to call it with the -k option without the string terminating in the middle. How can I run tests with string parameters that contain these symbols? I am currently creating a dict and supplying range(len(my_dict)) as the parameter so I can access these variables via index, but I would prefer to be able to directly enter them in the commandline. EDIT: The current suggestions are all great and already solve some of my problems. However, I'm still not sure how I would call singular tests if my test function looked like this (it has more than one entry as opposed to this minimal example): @pytest.mark.parametrize('input, expected', [ ( """ integer :: & my_var !< my comment """, {'my_var': 'my comment'} ) ]) def test_fetch_variable_definitions_multiline(input,expected): ... 

Answer: Answer to your question including "EDIT" section: You can use following syntax to run pytest pytest .\tests\test_package_1\test_module_1_1.py::TestClass111::test_1111["param_name"]  where param_name can be both value of single parameter or pytest param id To assign some id to parameter set you can use following syntax: class TestClass111: @pytest.mark.parametrize( "param_1_name, param_2_name", [ ("Some difficult param 1", {"a": 1}), ("Some difficult param 2", {"b": 2}), ], ids=["id1", "id2"] ) def test_1111(self, request, param_1_name, param_2_name):  or class TestClass111: @pytest.mark.parametrize( "param_1_name, param_2_name", [ pytest.param("Some difficult param 1", {"a": 1}, id="id1"), pytest.param("Some difficult param 2", {"b": 2}, id="id2"), ], ) def test_1111(self, request, param_1_name, param_2_name):  so you can access test with param set with id=id1 like this: pytest .\tests\test_package_1\test_module_1_1.py::TestClass111::test_1111["id1"]
Question: I'm trying to output a CSV file from Python and make one of the entries a Google sheet formula: This is what the formula var would look like:  strLink = "https://xxxxxxx.xxxxxx.com/Interact/Pages/Content/Document.aspx?id=" + strId + "&SearchId=0&utm_source=interact&utm_medium=general_search&utm_term=*" strLinkCellFormula = "=HYPERLINK(\"" + strLink + "\", \"" + strTitle + "\")"  and then for each row of the CSV I have this:  strCSV = strCSV + strId + ", " + "\"" + strTitle + "\", " + strAuthor + ", " + strDate + ", " + strStatus + ", " + "\"" + strSection + "\", \"" + strLinkCellFormula +"\"\n"  Which doesn't quite work, the hyperlink formula for Google sheets is like so: =HYPERLINK(url, title)  and I can't seem to get that comma escaped. So in my Sheet I am getting an additional column with the title in it and obviously the formula does not work.

Answer: Instead of reinventing the wheel, you should write your CSV rows using the builtin csv.writer class. This takes care of escaping any commas and quotes in the data, so you don't need to build your own escape logic. This helps you avoid the mess of escaping in your strLinkCellFormula = ... and strCSV = strCSV + ... lines. For example: import csv urls = ["https://google.com", "https://stackoverflow.com/", "https://www.python.org/"] titles = ["Google", "Stack Overflow", "Python"] with open("file.csv", "w") as fw: writer = csv.writer(fw) writer.writerow(["Company", "Website"]) for u, t in zip(urls, titles): formula = f'=HYPERLINK("{u}", "Visit {t}")' row = [t, formula] writer.writerow(row)  Note that in the line formula = ... above, I used the f-string syntax to format the URL and title into the string. I also used apostrophes to define the string, since I knew that the string was going to contain quotation marks and I didn't want to bother escaping them. This gives the following CSV: Company,Website Google,"=HYPERLINK(""https://google.com"", ""Visit Google"")" Stack Overflow,"=HYPERLINK(""https://stackoverflow.com/"", ""Visit Stack Overflow"")" Python,"=HYPERLINK(""https://www.python.org/"", ""Visit Python"")"  where the escaping of commas and quotes is already taken care of. It is also read by Excel/GSheets correctly, since it conforms to the standard CSV format:    For your specific case, you'd write to your CSV file like so: with open(filename, "w") as wf: writer = csv.writer(wf) writer.writerow(headers) # if necessary for ...: strLink = f"https://xxxxxxx.xxxxxx.com/Interact/Pages/Content/Document.aspx?id={strID}&SearchId=0&utm_source=interact&utm_medium=general_search&utm_term=*" strLinkCellFormula = f'=HYPERLINK("{strLink}", "{strTitle}")' row = [strId, strTitle, strAuthor, strDate, strStatus, strSection, strLinkCellFormula] writer.writerow(row)
Question: I'm trying to create a custom enumerator that can replace an int, but has additional fields. from enum import IntEnum class MD_Fields(IntEnum): ACCOUNT = (0, "Account", True) M_DESCRIPT = (4, "Description", False) def __new__(cls, value: int, description: str, identifier: bool): obj = int.__new__(cls, value) obj.description = description obj.identifier = identifier return obj if __name__ == '__main__': print(MD_Fields.M_DESCRIPT)  However, this code raises the following problem: Traceback (most recent call last): File ".../JetBrains/PyCharmCE2022.3/scratches/scratch.py", line 3, in <module> class MD_Fields(IntEnum): File "/usr/lib/python3.7/enum.py", line 223, in __new__ enum_member._value_ = member_type(*args) TypeError: int() takes at most 2 arguments (3 given)  I don't understand what's happening. (I didn't find a meaningful definition of int.__new__)

Answer: You were close - it was only missing the obj._value_ = value assignment, which Enum needs: from enum import IntEnum class MD_Fields(IntEnum): ACCOUNT = (0, "Account", True) M_DESCRIPT = (4, "Description", False) def __new__(cls, value: int, description: str, identifier: bool): obj = int.__new__(cls, value) obj._value_ = value obj.description = description obj.identifier = identifier return obj  After adding that, it works as intended: >>> for member in MD_Fields: ... member, int(member), member.description, member.identifier ... (<MD_Fields.ACCOUNT: 0>, 0, 'Account', True) (<MD_Fields.M_DESCRIPT: 4>, 4, 'Description', False)  If you were only extending Enum instead of IntEnum (which is just a shortcut for extending both int and Enum), you would use obj = object.__new__(cls) instead. The reason that _value_ needs to be set is because EnumMeta.__new__ will use its default behavior for setting _value_ when it wasn't already stored on the member. The relevant portion from the source for EnumMeta.__new__ (in 3.10), which should look familiar from the error that occurred before that attribute was assigned a value in MD_Fields.__new__: if not hasattr(enum_member, '_value_'): if member_type is object: enum_member._value_ = value else: enum_member._value_ = member_type(*args)  Normally, the ._value_ attribute would be populated using the value to the right of the = when the member was defined. That value ends up being exposed through the .value attribute for each member. In 3.11, the error was improved to include the hint that _value_ needs to be set: TypeError: _value_ not set in __new__, unable to create it
Question: Previously with setup.py you could just add setuptools.setup( ... scripts=[ "scripts/myscript.sh" ] )  and the shell script was just copied to the path of the environment. But with the new pyproject scpecification, this seems to not be possible any more. According to the Python specification of entry points and the setuptools specification, only python functions that will be wrapped later, are allowed. Does anyone know a simple way of doing this like in setup.py? Or at least simpler than just doing a python function that calls the shell script with subprocess, which is what I think I will do if there's no simpler way.

Answer: Probably using the script-files field of the [tool.setuptools] section should work: [tool.setuptools] script-files = ["scripts/myscript.sh"]  It was not standardized in PEP 621, so it belongs in a setuptools-specific section. Setuptools marks it as deprecated, but personally I would assume that it is safe to use for the next couple of years at least. It seems like such scripts are standardized in the wheel file format, so it is a bit strange that they are not in pyproject.toml's [project] section. Maybe it will be added later, but that is just speculation. Reference:  https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration https://packaging.python.org/en/latest/specifications/binary-distribution-format/ https://discuss.python.org/t/whats-the-status-of-scripts-vs-entry-points/18524
Question: I have a DataFrame with multiple columns and it looks like this: date col1 col2 col3 2023-01-01 Y N NaN 2023-01-02 Y N Y  Knowing the start and the end timestamp of df['date'], I want to generate the in between timestamps with the desired frequency. That I can do it using the code below: new_date = pd.Series(pd.date_range(start=df.index[0], end=df.index[-1], freq= '15min')) However, I would like to generate for the rest of the columns an equal number of rows with len(new_date) and fill the in between values with the value before. I know I can do that using .ffill() but I don't know how to generate the missing rows. The end result should look like this: date col1 col2 col3 2020-11-01 00:00:00 Y N NaN 2020-11-01 00:15:00 Y N NaN 2020-11-01 00:30:00 Y N NaN ... 2023-01-02 00:00:00 Y N Y 

Answer: You can reindex with those datetimes and forward fill the missings. This way, if the starting value was NaN, the followings will stay as NaN as well: >>> df.reindex(new_date, method="ffill") col1 col2 col3 2023-01-01 00:00:00 Y N NaN 2023-01-01 00:15:00 Y N NaN 2023-01-01 00:30:00 Y N NaN 2023-01-01 00:45:00 Y N NaN 2023-01-01 01:00:00 Y N NaN ... ... ... ... 2023-01-01 23:00:00 Y N NaN 2023-01-01 23:15:00 Y N NaN 2023-01-01 23:30:00 Y N NaN 2023-01-01 23:45:00 Y N NaN 2023-01-02 00:00:00 Y N Y [97 rows x 3 columns]  where new_date is pd.date_range(start=df.index[0], end=df.index[-1], freq="15min").
Question: This one when I run generates error qs = User.objects.annotate(days=(datetime.now() - F("created_at")).days)  AttributeError: 'CombinedExpression' object has no attribute 'days'  How can I make that calculation as an annotation When I run this code, it wroks fine qs = User.objects.annotate(days=(datetime.now() - F("created_at"))) 

Answer: This can be achieved with a combination of ExpressionWrapper, which tells django what the output field type should be, and ExtractDay which, well, extracts the day. In this case, the output field is a timedelta object (i.e DurationField). ExtractDay is just a DB-level function which the django ORM provides. from django.db.models import DateTimeField, DurationField, ExpressionWrapper, F from django.db.models.functions import ExtractDay qs = User.objects.annotate( days=ExtractDay( ExpressionWrapper( datetime.now() - F("created_at"), output_field=DurationField())) )
Question: I'm starting Pytorch and still trying to understand the basic concepts. If I have a network n on the GPU that produces an output tensor out, can it be printed to stdout directly? Or should it first be moved to the cpu, or be detached from the graph before printing? Tried several combinations below involving .cpu() and .detach() import torch.nn as nn import torch class Net(nn.Module): def __init__(self): super().__init__() self.layers = nn.Sequential( nn.Linear(5, 10), nn.ReLU(), nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 3), ) def forward(self, x): return self.layers(x) device = torch.device("cuda:0") # assume its available x = torch.rand(10, 5).to(device) net = Net().to(device) # Pretend we are in a training loop iteration out = net(x) print(f"The output is {out.max()}") print(f"The output is {out.max().detach()}") print(f"The output is {out.max().cpu()}") print(f"The output is {out.max().cpu().detach()}") # continue training iteration and repeat more iterations in training loop  I got the same output for all 4 methods. Which is the correct way?

Answer: You should not get surprised by the same value output. It shouldn't change anything value. cpu() transfers the tensor to cpu. And detach() detaches the tensor from the computation graph so that autograd does not track it for future backpropagations. Usually .detach().cpu() is what I do, since it detaches it from the computation graph and then it moves to the cpu for further processing. .cpu().detach() is also fine but in this case autograd takes into account the cpu() but in the previous case .cpu() operation won't be tracked by autograd which is what we want. That's it. It's only these little things that are different - value would be same in all cases.
Question: I cannot understand how to use previous indexes within an apply() or similar. This is the code: for i, row in data.iterrows(): index = data.index.get_loc(i) if index == 0: pass else: # changes data.at[i, '1_Day_%_Change'] = ( data.at[data.index[index], 'Adj_Close'] / data.at[data.index[index-1], 'Adj_Close'] ) - 1 data.at[i, '5_Day_%_Change'] = data.at[data.index[index], 'Adj_Close'] / data.at[data.index[index-5], 'Adj_Close'] - 1 data.at[i, '1_Month_%_Change'] = data.at[data.index[index], 'Adj_Close'] / data.at[data.index[index-21], 'Adj_Close'] - 1 data.at[i, '6_Monthr_%_Change'] = data.at[data.index[index], 'Adj_Close'] / data.at[data.index[index-151], 'Adj_Close'] - 1 data.at[i, '1_Year_%_Change'] = data.at[data.index[index], 'Adj_Close'] / data.at[data.index[index-252], 'Adj_Close'] - 1  data is the dataframe, and the goal is just to make % changes for stock prices. All I am doing is dividing the current row's 'Adj Close' price by the price X rows ago. How can I speed this up?

Answer: Use diff and shift methods. Example code is here. df['1_Day_%_Change'] = df['Adj_close'].diff() / df['Adj_close'].shift(1) df['5_Day_%_Change'] = df['Adj_close'].diff(5) / df['Adj_close'].shift(5)
Question: I'm new to Python, and I've been stuck at one point for several days now. There is a list of dicts like this one: dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] I need to output the value in prod based on the maximum value of the price. Here is the desired result: Green I have tried many ways based on information I found on SO: dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] L = [v for v in dd if v['price']==max([u['price']for u in dd])][0]['prod'] print(L)  Output: Blue (Almost correct, but "Blue" does not have the maximum value of the price!) dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] L = max(dd, key=lambda x:x['price']) print(L)  Output: {'prod': 'Blue', 'price': '9.718'} dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] L = max(e['price'] for e in dd) print(L)  Output: 9.718 from operator import itemgetter dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] L = max(map(itemgetter('price'), dd)) print(L)  Output: 9.718 dd = [{'prod': 'White', 'price': '80.496'}, {'prod': 'Blue', 'price': '9.718'}, {'prod': 'Green', 'price': '7161.3'}] seq = [x['price'] for x in dd] L = max(seq) print(L)  Output: 9.718 In all cases, the maximum value is 9.718 and not 7161.3. How can I fix this? I'm using MS Visual Studio running Python 3.9.

Answer: You need to convert the price values to floats for the key parameter: max(dd, key=lambda x: float(x['price']))['prod']  This outputs: Green
Question: When I want to import osmnx, this error comes up. I created a new environment before and followed the standard installation process via conda. 

Answer: You're using a years-old version of OSMnx and a brand new version of Shapely. They are incompatible.  OSMnx >= 1.3 works with Shapely >= 2.0, see here. OSMnx < 1.3 works with Shapely < 2.0, see here.  Recreate your environment (and optionally explicitly specify osmnx=1.3.*) and it'll work. Make sure you follow the documented installation instructions and honor the dependency versions specified in OSMnx's requirements.txt file.
Question: Is there a neat way to aggregate columns into a new column without duplicating information? For example, if I have a df:  Description Information 0 text1 text1 1 text2 text3 2 text4 text5  And I want to create a new column called 'Combined', which aggregates 'Description' and 'Information' to get:  Description Information Combined 0 text1 text1 text1 1 text2 text3 text2 text3 2 text4 text5 text4 text5  So far I have been using np.where and [mask] to check for duplicates before aggregating with df['Combined'] = df[['Description', 'Information']].agg(' '.join, axis=1) Although this works, it is not practical on a larger scale, grateful if anyone knows of a simpler way!

Answer: You can first run unique: df['Combined'] = (df[['Description', 'Information']] .agg(lambda x: ' '.join(x.unique()), axis=1) )  Output:  Description Information Combined 0 text1 text1 text1 1 text2 text3 text2 text3 2 text4 text5 text4 text5