eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My use case is to call fixture only if a certain condition is met. But since we need to call the pytest fixture as an argument to a test function it gets called every time I run the test. I want to do something like this: @pytest.parameterize("a", [1, 2, 3]) def test_method(a): if a == 2: method_fixture 

Answer: Yes, you can use indirect=True for a parameter to have the parameter refer to a fixture. import pytest @pytest.fixture def thing(request): if request.param == 2: return func() return None @pytest.mark.parametrize("thing", [1, 2, 3], indirect=True) def test_indirect(thing): pass # thing will either be the retval of `func()` or NOne  With dependent "fixtures" As asked in the edit, if your fixtures are dependent on each other, you'll probably need to use the pytest_generate_tests hook instead. E.g. this will parametrize the test with values that aren't equal. import itertools def pytest_generate_tests(metafunc): if metafunc.function.__name__ == "test_combo": a_values = [1, 2, 3, 4] b_values = [2, 3, 4, 5] all_combos = itertools.product(a_values, b_values) combos = [ pair for pair in all_combos if pair[0] != pair[1] ] metafunc.parametrize(["a", "b"], combos) def test_combo(a, b): assert a != b
Question: I want to inherit my dataclass but remove some of its fields. How can I do that in runtime so I don't need to copy all of the members one by one? Example: from dataclasses import dataclass @dataclass class A: a: int b: int c: int d: int @remove("c", "d") class B(A): pass  Such that A would have a, b, c, d defined and B would only have a and b defined.

Answer: We can remove the particular fields from the __annotations__ dictionary as well as from __dataclass_fields__ and then rebuild our class using dataclasses.make_dataclass: def remove(*fields): def _(cls): fields_copy = copy.copy(cls.__dataclass_fields__) annotations_copy = copy.deepcopy(cls.__annotations__) for field in fields: del fields_copy[field] del annotations_copy[field] d_cls = dataclasses.make_dataclass(cls.__name__, annotations_copy) d_cls.__dataclass_fields__ = fields_copy return d_cls return _  Note that we copy the annotations and the fields in order to not affect A. otherwise it would remove these fields from A as well and any other attempt to inherit A and remove again a field which we already removed would lead to an error. I.e: from dataclasses import dataclass @dataclass class A: a: int b: int c: int d: int @remove("c", "d") class B(A): pass @remove("b", "c") class C(A): pass  This would give us a KeyError since we already removed "c" and this is no longer exists in A's dictionary.
Question:  hovertemplate= 'Continent: %{df['continent']}'+ 'Country: %{df['country']}'+ 'gdpPercap: %{x:,.4f} '+ 'lifeExp: %{y}'+ '' I'm trying to use hovertemplate to customize hover information. However I can't get it to display what I want. I am getting x & y to work well. I can't figure out how to add other fields to the hovertemplate though. Any help would be appreciated. import numpy as np df = df[df['year'] == 1952] customdata = np.stack((df['continent'], df['country']), axis=-1) fig = go.Figure() for i in df['continent'].unique(): df_by_continent = df[df['continent'] == i] fig.add_trace(go.Scatter(x=df_by_continent['gdpPercap'], y=df_by_continent['lifeExp'], mode='markers', opacity=.7, marker = {'size':15}, name=i, hovertemplate= 'Continent: %{customdata[0]}<br>'+ 'Country: %{customdata[1]}<br>'+ 'gdpPercap: %{x:,.4f} <br>'+ 'lifeExp: %{y}'+ '<extra></extra>', )) fig.update_layout(title="My Plot", xaxis={'title':'GDP Per Cap', 'type':'log'}, yaxis={'title':'Life Expectancy'}, ) fig.show()  Updated with more code. The first answer didn't work just returning the text value of comdata.

Answer: See below for an additional example of how to use customdata with multiple traces based on the code included in your question. Note that you actually need to add the customdata to the figure traces in order to use it in the hovertemplate, this was also shown in Derek O's answer. import numpy as np import pandas as pd import plotly.graph_objects as go df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/gapminderDataFiveYear.csv') df = df[df['year'] == 1952] fig = go.Figure() for continent in df['continent'].unique(): df_by_continent = df[df['continent'] == continent] fig.add_trace( go.Scatter( x=df_by_continent['gdpPercap'], y=df_by_continent['lifeExp'], customdata=np.stack((df_by_continent['country'], df_by_continent['pop']), axis=-1), mode='markers', opacity=0.7, marker={'size': 15}, name=continent, hovertemplate='<b>Country</b>: %{customdata[0]}<br>' + '<b>Population</b>: %{customdata[1]:,.0f}<br>' + '<b>GDP</b>: %{x:$,.4f}<br>' + '<b>Life Expectancy</b>: %{y:,.2f} Years' + '<extra></extra>', ) ) fig.update_layout( xaxis={'title': 'GDP Per Cap', 'type': 'log'}, yaxis={'title': 'Life Expectancy'}, ) fig.write_html('fig.html', auto_open=True)
Question: Python has a library called Beautiful Soup that you can use to parse an HTML tree without creating 'get' requests in external web pages. I'm looking for the same in JavaScript, but I've only found jsdom and JSSoup (which seems unused) and if I'm correct, they only allow you to make requests. I want a library in JavaScript which allows me to parse the entire HTML tree without getting CORS policy errors, that is, without making a request, just parsing it. How can I do this?

Answer: In a browser context, you can use DOMParser: const html = "<h1>title</h1>"; const parser = new DOMParser(); const parsed = parser.parseFromString(html, "text/html"); console.log(parsed.firstChild.innerText); // "title"  and in node you can use node-html-parser: import { parse } from 'node-html-parser'; const html = "<h1>title</h1>"; const parsed = parse(html); console.log(parsed.firstChild.innerText); // "title"
Question: given these sublists lst=[['a', 'b', 'c', 'd', 'e'], ['f', 'g', 'h']]  I am trying to find the location of its elements, for instance, the letter 'a' is located at 0,0 but this line print(lst.index('a'))  instead produces the following error: ValueError: 'a' is not in list

Answer: You can use list comprehension: >>> lst=[['a', 'b', 'a', 'd', 'a'], ['f', 'g', 'a'], ['a','a','b']] >>> [(i,j) for i in range(len(lst)) for j in range(len(lst[i])) if lst[i][j]=='a'] [(0, 0), (0, 2), (0, 4), (1, 2), (2, 0), (2, 1)]
Question: In programming, I used only Integers. But this time for some calculations. I need to calculate Euler-Mascheroni Constant γ . up to n-th decimal.{Though n ∈ [30, 150] is enough for me.   [x] = gif(x) = math.floor(x)  But, I doubt the precision Numerical Algorithm I need higher degree of accuracy using Python.

Answer: From the French Wikipedia discussion page, an approximation to 6 decimal places: import math as m EulerMascheroniApp = round( (1.-m.gamma(1+1.e-8))*1.e14 )*1.e-6 print(EulerMascheroniApp) # 0.577216  This constant is also available in the sympy module, under the name EulerGamma: >>> import sympy >>> sympy.EulerGamma EulerGamma >>> sympy.EulerGamma.evalf() 0.577215664901533 >>> - sympy.polygamma(0,1) EulerGamma >>> sympy.stieltjes(0) EulerGamma >>> sympy.stieltjes(0, 1) EulerGamma  Documentation:  math.gamma; sympy.EulerGamma; sympy.functions.special; sympy: numerical evaluation.  On this last documentation link, you can find more information about how to evaluate the constant with more precision, if the default of .evalf() is not enough. If you still want to compute the constant yourself as an exercise, I suggest comparing your results to sympy's constant, to check for accuracy and correctness.
Question: I have a PySpark DataFrame and I would like to get the second highest value of ORDERED_TIME (DateTime Field yyyy-mm-dd format) after a groupBy applied to 2 columns, namely CUSTOMER_ID and ADDRESS_ID. A customer can have many orders associated with an address and I would like to get the second most recent order for a (customer,address) pair My approach was to make a window and partition according to CUSTOMER_ID and ADDRESS_ID, sort by ORDERED_TIME  sorted_order_times = Window.partitionBy("CUSTOMER_ID", "ADDRESS_ID").orderBy(col('ORDERED_TIME').desc()) df2 = df2.withColumn("second_recent_order", (df2.select("ORDERED_TIME").collect()[1]).over(sorted_order_times))  However, I get an error saying ValueError: 'over' is not in list Could anyone suggest the right way to go about solving this problem? Please let me know if any other information is needed Sample Data +-----------+----------+-------------------+ |USER_ID |ADDRESS_ID| ORDER DATE | +-----------+----------+-------------------+ | 100| 1000 |2021-01-02 | | 100| 1000 |2021-01-14 | | 100| 1000 |2021-01-03 | | 100| 1000 |2021-01-04 | | 101| 2000 |2020-05-07 | | 101| 2000 |2021-04-14 | +-----------+----------+-------------------+  Expected Output +-----------+----------+-------------------+-------------------+ |USER_ID |ADDRESS_ID| ORDER DATE |second_recent_order +-----------+----------+-------------------+-------------------+ | 100| 1000 |2021-01-02 |2021-01-04 | 100| 1000 |2021-01-14 |2021-01-04 | 100| 1000 |2021-01-03 |2021-01-04 | 100| 1000 |2021-01-04 |2021-01-04 | 101| 2000 |2020-05-07 |2020-05-07 | 101| 2000 |2021-04-14 |2020-05-07 +-----------+----------+-------------------+------------------- 

Answer: Here is another way to do it. Using collect_list import pyspark.sql.functions as F from pyspark.sql import Window sorted_order_times = Window.partitionBy("CUSTOMER_ID", "ADDRESS_ID").orderBy(F.col('ORDERED_TIME').desc()).rangeBetween(Window.unboundedPreceding, Window.unboundedFollowing) df2 = ( df .withColumn("second_recent_order", (F.collect_list(F.col("ORDERED_TIME")).over(sorted_order_times))[1]) ) df2.show()
Question: I use plotly dash to draw the following scatter charts, and how can I change the grid line color 

Answer: To customize the grid in plotly, do the following. This will allow you to display the grid on the xy axis, set the line width and line color. fig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='LightPink') fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='LightPink')
Question: Assuming the following: >>> square = '²' # Superscript Two (Unicode U+00B2) >>> cube = '³' # Superscript Three (Unicode U+00B3)  Curiously: >>> square.isdigit() True >>> cube.isdigit() True  OK, let's convert those "digits" to integer: >>> int(square) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: invalid literal for int() with base 10: '²' >>> int(cube) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: invalid literal for int() with base 10: '³'  Oooops! Could someone please explain what behavior I should expect from the str.isdigit() method when handling strings?

Answer: str.isdigit doesn't claim to be related to parsability as an int. It's reporting a simple Unicode property, is it a decimal character or digit of some sort:  str.isdigit() Return True if all characters in the string are digits and there is at least one character, False otherwise. Digits include decimal characters and digits that need special handling, such as the compatibility superscript digits. This covers digits which cannot be used to form numbers in base 10, like the Kharosthi numbers. Formally, a digit is a character that has the property value Numeric_Type=Digit or Numeric_Type=Decimal.  In short, str.isdigit is thoroughly useless for detecting valid numbers. The correct solution to checking if a given string is a legal integer is to call int on it, and catch the ValueError if it's not a legal integer. Anything else you do will be (badly) reinventing the same tests the actual parsing code in int() performs, so why not let it do the work in the first place? Side-note: You're using the term "utf-8" incorrectly. UTF-8 is a specific way of encoding Unicode, and only applies to raw binary data. Python's str is an "idealized" Unicode text type; it has no encoding (under the hood, it's stored encoded as one of ASCII, latin-1, UCS-2, UCS-4, and possibly also UTF-8, but none of that is visible at the Python layer outside of indirect measurements like sys.getsizeof, which only hints at the underlying encoding by letting you see how much memory the string consumes). The characters you're talking about are simple Unicode characters above the ASCII range, they're not specifically UTF-8.
Question: enter image description here i can not solve .... how to solve : TypeError: the JSON object must be str, bytes or bytearray, not TextIOWrapper

Answer: You are using loads when you need load. json.load is for file-like objects, and json.loads is for strings. (You could also load the string into memory and then parse it with json.load, but you don't want to do that).
Question: I am trying to mock an AWS SQS with moto, below is my code from myClass import get_msg_from_sqs from moto import mock_sqs #from moto.sqs import mock_sqs @mock_sqs def test_get_all_msg_from_queue(): #from myClass import get_msg_from_sqs conn = boto3.client('sqs', region_name='us-east-1') queue = conn.create_queue(QueueName='Test') os.environ["SQS_URL"] = queue["QueueUrl"] queue.send_message( MessageBody=json.dumps({'a': '1', 'b': '2', 'c': '3'})) #Tried this as well #conn.send_message(QueueUrl=queue["QueueUrl"], MessageBody=json.dumps({'a': '1', 'b': '2', 'c': '3'})) resp = get_msg_from_sqs(queue["QueueUrl"]) assert resp is not None  While executing this I am getting the following error > queue.send_message( MessageBody=json.dumps({'a': '1', 'b': '2', 'c': '3'})) E AttributeError: 'dict' object has no attribute 'send_message'  If I try another way of to send a message in SQS(see commented out code #Tried this as well) then at the time of actual SQS calling in my method get_msg_from_sqs, I am getting the below error E botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.  I am running it on win10 with PyCharm and the moto version is set to moto = "^2.2.6"  My code is given below sqs = boto3.client('sqs') def get_msg_from_queue(queue_url: str) -> dict: return sqs.receive_message(QueueUrl=queue_url, AttributeNames=['All'], MaxNumberOfMessages=1, VisibilityTimeout=3600, WaitTimeSeconds=0)  What am I missing over here?

Answer: Your queue variable is a dict returned by create_queue: queue = conn.create_queue(QueueName='Test')  It is not a queue and thus you cannot call sendMessage on it. To do that, you need to create a queue object: conn = boto3.client('sqs') sqs = boto3.resource('sqs') response = conn.create_queue(QueueName='Test') queue_url = response["QueueUrl"] queue = sqs.Queue(queue_url) queue.send_message()
Question: I am trying to execute the following code for a nlp proj import torchtext from torchtext.legacy.data import Field, BucketIterator, Iterator from torchtext.legacy import data ----> 6 from torchtext.legacy.data import Field, BucketIterator, Iterator 7 from torchtext.legacy import data 8 ModuleNotFoundError: No module named 'torchtext.legacy'.  I have tried it on both kaggle notebook and jupyter notebook and found the same error in both. i even tried to install !pip install -qqq deepmatcher==0.1.1 in kaggle to solve the issue but it still gives the same error. is there any solution to this?

Answer: Before you import torchtext.legacy, you need to !pip install torchtext==0.10.0. Maybe legacy was removed in version 0.11.0.
Question: I'm currently using isort --profile=black --line-length=79 as a linter in my project for python files. This produces the Vertical Hanging Indent (mode 3 in isort's documentation kind of output: from third_party import ( lib1, lib2, lib3, lib4, )  This multiline mode only applies if the line is longer than 79 characters, though. Is there a mode that cause a multiline output as soon as there are two or more imports on the same line, no matter how long the line is? I tried hacking it with  isort -m=3 --trailing-comma --line-length=1, but shorter line length will cause multiline output even when there is a single import, which I don't want: from third_party import ( lib1, ) 

Answer: You should use the --force-grid-wrap 2 flag in the CLI or set in the settings file like pyproject.toml option force_grid_wrap = 2. This would force isort to produce multiline output for 2 or more imports, regardless of line length. More info about this option
Question: I currently use this strategy when I cannot assign default arguments in a function's signature and/or None already has meaning. from typing import Optional DEFAULT = object() # `None` already has meaning. def spam(ham: Optional[list[str]] = DEFAULT): if ham is DEFAULT: ham = ['prosciutto', 'jamon'] if ham is None: print('Eggs?') else: print(str(len(ham)) + ' ham(s).')  Error: Failed (exit code: 1) (2607 ms) main.py:7: error: Incompatible default for argument "ham" (default has type "object", argument has type "Optional[List[str]]") Found 1 error in 1 file (checked 1 source file)   How do I type-hint ham without getting errors in mypy? or What strategy should I use instead of DEFAULT = object()? 

Answer: Something I like to do — which is only a slight variation on @Blckknght's answer — is to use a metaclass to give my sentinel class a nicer repr and make it always-falsey.  sentinel.py from typing import Literal class SentinelMeta(type): def __repr__(cls) -> str: return f'<{cls.__name__}>' def __bool__(cls) -> Literal[False]: return False class Sentinel(metaclass=SentinelMeta): pass  main.py from sentinel import Sentinel class DEFAULT(Sentinel): pass   You use it in type hints exactly in the same way @Blckknght suggests: def spam(ham: list[str]|None|type[DEFAULT] = DEFAULT): ...  But you have the added advantages that your sentinel value is always falsey and has a nicer repr: >>> DEFAULT <DEFAULT> >>> bool(DEFAULT) False
Question: I noticed that for string in Dataframe will keep left align in Excel and for numerical value will keep right align in Excel. How do we set the desired alignment we wanted when exporting DataFrame to Excel? Example: Center Alignment df = pd.DataFrame({"colname1": ["a","b","c","d"], "colname2": [1,2,3,4]}) with pd.ExcelWriter("test.xlsx") as writer: df.to_excel( writer, index=False, header=False, ) 

Answer: You can set the styles of the dataframe using the Styler object, which uses the same conventions as CSS. The documentation has a great primer on the different ways of styling your dataframes. For a simple solution to your example, you can set the desired alignment by first creating a function: def align_center(x): return ['text-align: center' for x in x]  Then write it to Excel while applying the function you just defined: with pd.ExcelWriter("test.xlsx") as writer: df.style.apply(align_center, axis=0).to_excel( writer, index=False, header=False )  This will center-align the cells in the Excel file. For an exhaustive list of available text alignment options I would suggest the MDN docs.
Question: df  ts_code type close 0 861001.TI 1 648.399 1 861001.TI 20 588.574 2 861001.TI 30 621.926 3 861001.TI 60 760.623 4 861001.TI 90 682.313 ... ... ... ... 8328 885933.TI 5 1083.141 8329 885934.TI 1 951.493 8330 885934.TI 5 1011.346 8331 885935.TI 1 1086.558 8332 885935.TI 5 1028.449  Goal ts_code l5d_close l20d_close …… l90d_close 861001.TI NaN 1.10 0.95 …… …… …… ……  I want to groupby ts_code to calculate the close of type(1)/the close of type(N:5,20,30……). Take 861001.TI for example, l5d_close is nan because there is no value when the type is 5. l20d_close equals 648.399/588.574=1.10, l90d_close equals 648.399/682.313=0.95. And the result is rounded. Try df.groupby('ts_code')\ .pipe(lambda x: x[x.type==1].close/x[x.type==10].close) Got: KeyError: 'Column not found: False'  The type values is: 1,5,20,30,60,90,180,200 Notice: There is one value of type columns for each ts_code

Answer: Use sort_values to make sure type == 1 is the first row per group and extract them with groupby.transform('first'): df = df.sort_values(['ts_code', 'type']) close1 = df.groupby('ts_code')['close'].transform('first') df['close'] = close1 / df['close'] # ts_code type close # 0 861001.TI 1 1.000000 # 1 861001.TI 20 1.101644 # 2 861001.TI 30 1.042566 # 3 861001.TI 60 0.852458 # ... ... ... ...  Then pivot the type column into column headers: out = (df.pivot(index='ts_code', columns='type', values='close') .drop(columns=1) .add_prefix('l') .add_suffix('d_close')) # type l5d_close l20d_close l30d_close l60d_close l90d_close # ts_code # 861001.TI NaN 1.101644 1.042566 0.852458 0.950296 # ... ... ... ... ... ...   To chain together, assign a ratio column before the pivot: (df.assign(ratio=df.groupby('ts_code').close.transform('first').div(df.close)) .pivot(index='ts_code', columns='type', values='ratio') .drop(columns=1) .add_prefix('l') .add_suffix('d_close')) # type l5d_close l20d_close l30d_close l60d_close l90d_close # ts_code # 861001.TI NaN 1.101644 1.042566 0.852458 0.950296 # ... ... ... ... ... ...
Question: Following gensim word2vec embedding tutorial, I have trained a simple word2vec model: from gensim.test.utils import common_texts from gensim.models import Word2Vec model = Word2Vec(sentences=common_texts, size=100, window=5, min_count=1, workers=4) model.save("/content/word2vec.model")  I would like to visualize it using the Embedding Projector in TensorBoard. There is another straightforward tutorial in gensim documentation. I did the following in Colab: !python3 -m gensim.scripts.word2vec2tensor -i /content/word2vec.model -o /content/my_model Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/dist-packages/gensim/scripts/word2vec2tensor.py", line 94, in <module> word2vec2tensor(args.input, args.output, args.binary) File "/usr/local/lib/python3.7/dist-packages/gensim/scripts/word2vec2tensor.py", line 68, in word2vec2tensor model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_model_path, binary=binary) File "/usr/local/lib/python3.7/dist-packages/gensim/models/keyedvectors.py", line 1438, in load_word2vec_format limit=limit, datatype=datatype) File "/usr/local/lib/python3.7/dist-packages/gensim/models/utils_any2vec.py", line 172, in _load_word2vec_format header = utils.to_unicode(fin.readline(), encoding=encoding) File "/usr/local/lib/python3.7/dist-packages/gensim/utils.py", line 355, in any2unicode return unicode(text, encoding, errors=errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte  Please note that I did check first this exact same question from 2018 - but the accepted answer no longer works as both in gensim and tensorflow have been updated so I considered it was worth asking again in Q4 2021.

Answer: Saving the model in the original C word2vec implementation format resolves the issue: model.wv.save_word2vec_format("/content/word2vec.model"): from gensim.test.utils import common_texts from gensim.models import Word2Vec model = Word2Vec(sentences=common_texts, size=100, window=5, min_count=1, workers=4) model.wv.save_word2vec_format("/content/word2vec.model")  There are two formats of storing word2vec models in gensim: keyed vector format from the original word2vec implementation and format that additionally stores hidden weights, vocabulary frequencies, and more. Examples and details can be found in the documentation. The script word2vec2tensor.py uses the original format and loads the model with load_word2vec_format: code.