eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: This question is very similar to one I posted before with just one change. Instead of doing just the absolute difference for all the columns I also want to find the magnitude difference for the 'Z' column, so if the current Z is 1.1x greater than prev than keep it. (more context to the problem) Pandas using the previous rank values to filter out current row df = pd.DataFrame({ 'rank': [1, 1, 2, 2, 3, 3], 'x': [0, 3, 0, 3, 4, 2], 'y': [0, 4, 0, 4, 5, 5], 'z': [1, 3, 1.2, 3.25, 3, 6], }) print(df) # rank x y z # 0 1 0 0 1.00 # 1 1 3 4 3.00 # 2 2 0 0 1.20 # 3 2 3 4 3.25 # 4 3 4 5 3.00 # 5 3 2 5 6.00  Here's what I want the output to be output = pd.DataFrame({ 'rank': [1, 1, 2, 3], 'x': [0, 3, 0, 2], 'y': [0, 4, 0, 5], 'z': [1, 3, 1.2, 6], }) print(output) # rank x y z # 0 1 0 0 1.0 # 1 1 3 4 3.0 # 2 2 0 0 1.2 # 5 3 2 5 6.00  basically what I want to happen is if the previous rank has any rows with x, y (+- 1 both ways) AND z (<1.1z) to remove it. So for the rows rank 1 ANY rows in rank 2 that have any combo of x = (-1-1), y = (-1-1), z= (<1.1) OR x = (2-5), y = (3-5), z= (<3.3) I want it to be removed

Answer: I have modified mozway's function so that it works according to your requirements. # comparing 'equal' float values, may go wrong, that's why I am using this constant DELTA=0.1**12 def check_previous_group(rank, d, groups): if not rank-1 in groups.groups: # check if a previous group exists, else flag all rows False (i.e. not to be dropped) #return pd.Series(False, index=d.index) return pd.Series(False, index=d.index) else: # get previous group (rank-1) d_prev = groups.get_group(rank-1) # get the absolute difference per row with the whole dataset # of the previous group: abs(d_prev-s) # if differences in x and y are within 1 and z < 1.1*x # for at least one row of the previous group # then flag the row to be dropped (True) return d.apply(lambda s: (abs(d_prev-s)[['x', 'y']].le([1,1]).all(1)& (s['z']<1.1*d_prev['x']-DELTA)).any(), axis=1)  tests, >>> df = pd.DataFrame({ 'rank': [1, 1, 2, 2, 3, 3], 'x': [0, 3, 0, 3, 4, 2], 'y': [0, 4, 0, 4, 5, 5], 'z': [1, 3, 1.2, 3.25, 3, 6], }) >>> df rank x y z 0 1 0 0 1.00 1 1 3 4 3.00 2 2 0 0 1.20 3 2 3 4 3.25 4 3 4 5 3.00 5 3 2 5 6.00 >>> groups = df.groupby('rank') >>> mask = pd.concat([check_previous_group(rank, d, groups) for rank,d in groups]) >>> df[~mask] rank x y z 0 1 0 0 1.0 1 1 3 4 3.0 2 2 0 0 1.2 5 3 2 5 6.0  >>> df = pd.DataFrame({ 'rank': [1, 1, 2, 2, 3, 3], 'x': [0, 3, 0, 3, 4, 2], 'y': [0, 4, 0, 4, 5, 5], 'z': [1, 3, 1.2, 3.3, 3, 6], }) >>> df rank x y z 0 1 0 0 1.0 1 1 3 4 3.0 2 2 0 0 1.2 3 2 3 4 3.3 4 3 4 5 3.0 5 3 2 5 6.0 >>> groups = df.groupby('rank') >>> mask = pd.concat([check_previous_group(rank, d, groups) for rank,d in groups]) >>> df[~mask] rank x y z 0 1 0 0 1.0 1 1 3 4 3.0 2 2 0 0 1.2 3 2 3 4 3.3 5 3 2 5 6.0
Question: We are using poetry to upgrade packages and deploy to our servers but some issue is stopping us from deploying our work continuously to our servers.The code below is the stacktrack where our code stops. $ poetry update Creating virtualenv kpbackend-ad2VTdyQ-py3.9 in /root/.cache/pypoetry/virtualenvs Updating dependencies Resolving dependencies... Writing lock file Package operations: 171 installs, 0 updates, 0 removals • Installing shortuuid (1.0.1) • Installing simpleeval (0.9.10) • Installing starkbank-ecdsa (1.1.1) • Installing starlette (0.13.8) EnvCommandError Command ['/root/.cache/pypoetry/virtualenvs/kpbackend-ad2VTdyQ-py3.9/bin/pip', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/49/c9/a8/c45627062eb893ac0685ce1146f6b868eea117d5803fc63c56d21326de/simpleeval-0.9.10.tar.gz'] errored with the following return code 1, and output: Processing /root/.cache/pypoetry/artifacts/49/c9/a8/c45627062eb893ac0685ce1146f6b868eea117d5803fc63c56d21326de/simpleeval-0.9.10.tar.gz ERROR: Command errored out with exit status 1: command: /root/.cache/pypoetry/virtualenvs/kpbackend-ad2VTdyQ-py3.9/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-408xzy97/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-408xzy97/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-lfjp6ggy cwd: /tmp/pip-req-build-408xzy97/ Complete output (1 lines): error in simpleeval setup command: use_2to3 is invalid. ---------------------------------------- WARNING: Discarding file:///root/.cache/pypoetry/artifacts/49/c9/a8/c45627062eb893ac0685ce1146f6b868eea117d5803fc63c56d21326de/simpleeval-0.9.10.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1180 in _run 1176│ output = subprocess.check_output( 1177│ cmd, stderr=subprocess.STDOUT, **kwargs 1178│ ) 1179│ except CalledProcessError as e: → 1180│ raise EnvCommandError(e, input=input_) 1181│ 1182│ return decode(output) 1183│ 1184│ def execute(self, bin, *args, **kwargs): Cleaning up file-based variables ERROR: Job failed: exit code 1  I am suspecting some package update broke it as 3 days ago this step worked fine in the Gitlab CI.

Answer: I encountered the same error some time ago. The issue seems to be connected with the latest upgrade of setuptools package (https://setuptools.pypa.io/en/latest/history.html#v58-0-0). The workaround that is working for me is to use setuptools<=57.5.0.
Question: Currently, I am checking for tuples with multiple (e.g. three) numbers of arbitrary but equal type in the following form: from typing import Tuple, Union Union[Tuple[int, int, int], Tuple[float, float, float]]  I want to make this check more generic, also allowing numpy number types. I.e. I tried to use numbers.Number: from numbers import Number from typing import Tuple Tuple[Number, Number, Number]  The above snipped also allows tuples of mixed types as long as everything is a number. I'd like to restrict the tuple to numbers of equal type. How can this be achieved?  Technically, this question applies to Python and the type hints specification itself. However, as pointed out in the comments, its handling is implementation specific, i.e. MyPy will not catch every edge case and/or inconsistency. Personally, I am using run-time checks with typeguard for testing and deactivate them entirely in production.

Answer: You can use TypeVar with bound argument. It allows restricting types to subtypes of a given type. In your case, the types should be the subtypes of Number: from numbers import Number from typing import TypeVar T = TypeVar('T', bound=Number) Tuple[T, T, T]  Why does it work? TypeVar is a variable that allows to use a particular type several times in type signatures. The simplest example: from typing import TypeVar, Tuple T = TypeVar('T') R = TypeVar('R') def swap(x: T, y: R) -> Tuple[R, T]: return y, x  The static type checker will infer that arguments of the function swap should be the same as outputs in reversed order (note that return type T will be the same as the input type T). Next, it might be useful to introduce restrictions for TypeVar. For instance, one could restrict values of a type variable to specific types: TypeVar('T', int, str). In this answer, we use another kind of restriction with a keyword bound – it checks if the values of the type variable are subtypes of a given type (in our case, Number, which is a base class for all numerical types in Python). More examples: mypy documentation and PEP 484.  Working test: from numbers import Number from typing import Tuple, TypeVar import numpy as np from typeguard import typechecked T = TypeVar('T', bound=Number) @typechecked def foo(data: Tuple[T, T, T]): print('expected OK:', data) for data in ( (1, 2, 3), # ok (1.0, 2.0, 3.0), # ok (1, 2.0, 3), # TypeError (1.0, 2.0, 3), # TypeError (1.0, 2.0, np.float64(3.0)), # ok (yes!) (1, 2.0, np.float32(3)), # TypeError (1, 2.0, np.uint8(3)), # TypeError ): try: foo( data ) except TypeError: print('expected TypeError:', data)
Question: I have the following panel stored in df:      state district year y constant x1 x2 time     0 01 01001 2009 12 1 0.956007 639673 1   1 01 01001 2010 20 1 0.972175 639673 2   2 01 01001 2011 22 1 0.988343 639673 3   3 01 01002 2009 0 1 0 33746 1   4 01 01002 2010 1 1 0.225071 33746 2   5 01 01002 2011 5 1 0.450142 33746 3   6 01 01003 2009 0 1 0 45196 1   7 01 01003 2010 5 1 0.427477 45196 2   8 01 01003 2011 9 1 0.854955 45196 3      y is the number of protests in each district constant is a column full of ones x1 is the proportion of the district's area covered by a mobile network provider x2 is the population count in each district (note that it is fixed in time)  How can I run the following model in Python?   Here's what I tried # Transform `x2` to match model df['x2'] = df['x2'].multiply(df['time'], axis=0) # District fixed effects df['delta'] = pd.Categorical(df['district']) # State-time fixed effects df['eta'] = pd.Categorical(df['state'] + df['year'].astype(str)) # Set indexes df.set_index(['district','year']) from linearmodels.panel import PanelOLS m = PanelOLS(dependent=df['y'], exog=df[['constant','x1','x2','delta','eta']])   ValueError: exog does not have full column rank. If you wish to proceed with model estimation irrespective of the numerical accuracy of coefficient estimates, you can set rank_check=False.  What am I doing wrong?

Answer: I dug around the documentation and the solution turned out to be quite simple. After setting the indexes and turning the fixed effect columns to pandas.Categorical types (see question above): # Import model from linearmodels.panel import PanelOLS # Model m = PanelOLS(dependent=df['y'], exog=df[['constant','x1','x2']], entity_effects=True, time_effects=False, other_effects=df['eta']) m.fit(cov_type='clustered', cluster_entity=True)  That is, DO NOT pass your fixed effect columns to exog. You should pass them to entity_effects (boolean), time_effects (boolean) or other_effects (pandas.Categorical).
Question: I'm writing a inference script for already trained NER model, but I have trouble with converting encoded tokens (their ids) into original words. # example input df = pd.DataFrame({'_id': [1], 'body': ['Amazon and Tesla are currently the best picks out there!']}) # calling method that handles inference: ner_model = NER() ner_model.recognize_from_df(df, 'body') # here is only part of larger NER class that handles the inference: def recognize_from_df(self, df: pd.DataFrame, input_col: str): predictions = [] df = df[['_id', input_col]].copy() dataset = Dataset.from_pandas(df) # tokenization, padding, truncation: encoded_dataset = dataset.map(lambda examples: self.bert_tokenizer(examples[input_col], padding='max_length', truncation=True, max_length=512), batched=True) encoded_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'], device=device) dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32) encoded_dataset_ids = encoded_dataset['_id'] for batch in dataloader: output = self.model(**batch) # decoding predictions and tokens for i in range(batch['input_ids'].shape[0]): tags = [self.unique_labels[label_id] for label_id in output[i]] tokens = [t for t in self.bert_tokenizer.convert_ids_to_tokens(batch['input_ids'][i]) if t != '[PAD]'] ...  The results are close to what I need: # tokens: ['[CLS]', 'am', '##az', '##on', 'and', 'te', '##sla', 'are', 'currently', 'the', 'best', 'picks', 'out', 'there', ...] # tags: ['X', 'B-COMPANY', 'X', 'X', 'O', 'B-COMPANY', 'X', 'O', 'O', 'O', 'O', 'O', 'O', 'O', ...]  How to combine 'am', '##az', '##on' and 'B-COMPANY', 'X', 'X' into one token/tag? I know that there is a method called convert_tokens_to_string in Tokenizer, but it returns just one big string, which is hard to map to tag. Regards

Answer: Provided you only want to "merge" company names one could do that in a linear time with pure Python. Skipping the beginning of sentence token [CLS] for brevity: tokens = tokens[1:] tags = tags[1:]  The function below will merge company tokens and increase pointer appropriately: def merge_company(tokens, tags): generated_tokens = [] i = 0 while i < len(tags): if tags[i] == "B-COMPANY": company_token = [tokens[i]] for j in range(i + 1, len(tags)): i += 1 if tags[j] != "X": break else: company_token.append(tokens[j][2:]) generated_tokens.append("".join(company_token)) else: generated_tokens.append(tokens[i]) i += 1 return generated_tokens  Usage is pretty simple, please notice tags need their Xs removed as well though: tokens = merge_company(tokens, tags) tags = [tag for tag in tags if tag != "X"]  This would give you: ['amazon', 'and', 'tesla', 'are', 'currently', 'the', 'best', 'picks', 'out', 'there'] ['B-COMPANY', 'O', 'B-COMPANY', 'O', 'O', 'O', 'O', 'O', 'O', 'O']
Question: I was checking the code of the toolz library's groupby function in Python and I found this: def groupby(key, seq): """ Group a collection by a key function """ if not callable(key): key = getter(key) d = collections.defaultdict(lambda: [].append) for item in seq: d[key(item)](item) rv = {} for k, v in d.items(): rv[k] = v.__self__ return rv  Is there any reason to use rv[k] = v.__self__ instead of rv[k] = v?

Answer: This is a somewhat confusing trick to save a small amount of time: We are creating a defaultdict with a factory function that returns a bound append method of a new list instance with [].append. Then we can just do d[key(item)](item) instead of d[key(item)].append(item) like we would have if we create a defaultdict that contains lists. If we don't lookup append everytime, we gain a small amount of time. But now the dict contains bound methods instead of the lists, so we have to get the original list instance back via __self__. __self__ is an attribute described for instance methods that returns the original instance. You can verify that with this for example: >>> a = [] >>> a.append.__self__ is a True
Question: I'm learning how to use SQL Alchemy, and I'm trying to re-implement a previously defined API but now using Python. The REST API has the following query parameter: myService/v1/data?range=time:2015-08-01:2015-08-02  So I want to map something like field:FROM:TO to filter a range of results, like a date range, for example. This is what I'm using at this moment:  rangeStatement = range.split(':') if(len(rangeStatement)==3): query = query.filter(text('{} BETWEEN "{}" AND "{}"'.format(*rangeStatement)))  So, this will produce the following WHERE condition: WHERE time BETWEEN "2015-08-01" AND "2015-08-02"  I know SQL Alchemy is a powerful tool that allows creating queries like Query.filter_by(MyClass.temp), but I need the API request to be as open as possible. So, I'm worried that someone could pass something like DROP TABLE in the range parameter and exploit the text function

Answer: If queries are constructed using string formatting then sqlalchemy.text will not prevent SQL injection - the "injection" will already be present in the query text. However it's not difficult to build queries dynamically, in this case by using getattr to get a reference to the column. Assuming that you are using the ORM layer with model class Foo and table foos you can do import sqlalchemy as sa ... col, lower, upper = 'time:2015-08-01:2015-08-02'.split(':') # Regardless of style, queries implement a fluent interface, # so they can be built iteratively # Classic/1.x style q1 = session.query(Foo) q1 = q1.filter(getattr(Foo, col).between(lower, upper)) print(q1)  or # 2.0 style (available in v1.4+) q2 = sa.select(Foo) q2 = q2.where(getattr(Foo, col).between(lower, upper)) print(q2)  The respective outputs are (parameters will be bound at execution time): SELECT foos.id AS foos_id, foos.time AS foos_time FROM foos WHERE foos.time BETWEEN ? AND ?  and SELECT foos.id, foos.time FROM foos WHERE foos.time BETWEEN :time_1 AND :time_2  SQLAlchemy will delegate quoting of the values to the connector package being used by the engine, so you're protection against injection will be as good as that provided by the connector package*.  * In general I believe correct quoting should be a good defence against SQL injections, however I'm not sufficiently expert to confidently state that it's 100% effective. It will be more effective than building queries from strings though.
Question: I'm working on a chart using Altair, and I'm trying to figure out how to have less lines in the background grid. Is there a term for that background grid? Here's a chart that looks like mine, that I took from the tutorial:  Let's say that I want to have half as many grid lines on the X axis. How could I do that?

Answer: Grid lines are drawn at the location of ticks, so to adjust the grid lines you can adjust the ticks. For example: import altair as alt import numpy as np import pandas as pd x = np.arange(100) source = pd.DataFrame({ 'x': x, 'f(x)': np.sin(x / 5) }) alt.Chart(source).mark_line().encode( x=alt.X('x', axis=alt.Axis(tickCount=4)), y='f(x)' )   You can see other tick-related properties in the documentation for alt.Axis.
Question: I'd like to dynamically create a function in memory, with a type hint of the argument. I do have some working code, but it feels extremely hacky and fragile. import typing func_name = 'some_function_name' req_type=int a = None exec(f'''def {func_name}(my_argument:{req_type.__name__}): pass a = {func_name}''') print(a) # <function some_function_name at 0x000001B89F3311F0> print(typing.get_type_hints(a)) # {'my_argument': <class 'int'>}  There has got to be a better way to do this. Any suggestions?

Answer: You can use FunctionType to create new functions. You can copy a template function and change its type hints and name. You also can change the code of the function object with compile Python function. I made an example that copy and change the name and the type hint of template function (without changing code) import types import typing def copy_func(f, func_types, name=None): # add your code to first parameter new_func = types.FunctionType(f.__code__, f.__globals__, name or f.__name__, f.__defaults__, f.__closure__) new_func.__annotations__ = func_types return new_func def template(arg): print('called template func') a = copy_func(template, {'my_argument': int}, "test") a(2) # can call it print("types:", typing.get_type_hints(a)) # types: {'my_argument': <class 'type'>}
Question: Let's say I have two dictionaries and I know want to measure the time needed to check if a key is in the dictionary. I tried to run this piece of code: from timeit import timeit dct1 = {str(i): 1 for i in range(10**7)} dct2 = {i: 1 for i in range(10**7)} print(timeit('"7" in dct1', setup='from __main__ import dct1', number=10**8)) print(timeit('7 in dct2', setup='from __main__ import dct2', number=10**8))  Here are the results that I get: 2.529034548999334 2.212983401999736  Now, let's say I try to mix integers and strings in both dictionaries, and measure access time again: dct1[7] = 1 dct2["7"] = 1 print(timeit('"7" in dct1', setup='from __main__ import dct1', number=10**8)) print(timeit('7 in dct1', setup='from __main__ import dct1', number=10**8)) print(timeit('7 in dct2', setup='from __main__ import dct2', number=10**8)) print(timeit('"7" in dct2', setup='from __main__ import dct2', number=10**8))  I get something weird: 3.443614432000686 2.6335261530002754 2.1873921409987815 2.272667104998618  The first value is much higher than what I had before (3.44 vs 2.52). However, the third value is basically the same as before (2.18 vs 2.21). Why is this happening? Can you reproduce the same thing or is this only me? Also, I can't understand the big difference between the first and the second value: it looks like it's more difficult to access a string key, but the same thing seems to apply only slightly to the second dictionary. Why? Update You don't even need to actually add a new key. All you need to do to see an increase in complexity is just checking if a key with different type exists!! This is much weirder than I thought. Look at the example here: from timeit import timeit dct1 = {str(i): 1 for i in range(10**7)} dct2 = {i: 1 for i in range(10**7)} print(timeit('"7" in dct1', setup='from __main__ import dct1', number=10**8)) # 2.55 print(timeit('7 in dct2', setup='from __main__ import dct2', number=10**8)) # 2.26 7 in dct1 "7" in dct2 print(timeit('"7" in dct1', setup='from __main__ import dct1', number=10**8)) # 3.34 print(timeit('7 in dct2', setup='from __main__ import dct2', number=10**8)) # 2.35 

Answer: Let me try to answer my own question. The dict implementation in CPython is optimised for lookups of str keys. Indeed, there are two different functions that are used to perform lookups:  lookdict is a generic dictionary lookup function that is used with all types of keys lookdict_unicode is a specialised lookup function used for dictionaries composed of str-only keys  Python will use the string-optimised version until a search for non-string data, after which the more general function is used. And it looks like you cannot even reverse the behaviour of a particular dict instance: once it starts using the generic function, you can't go back to using the specialised one!
Question: I always wanted to know if there is any real-world application of Pascal's triangle than just coefficients of the binomial expansion. I tried to solve this problem:   But, What if I am adding K units of water and wants to find a glass which has the least water in it: Where, Glass will be found as: c-th glass in r-th row And I believe. If we could find this then it will not be difficult for us to find amount of water in any glass for which {i<r and j<c}  Problem: Input Water added- K units and capacity of each glass as 1- unit Output Expected : c th glass in r th row having least water in it.   I tried to solve problem by keeping note of capacity of each row when it starts overflow: and wants to know how to keep going with this method.  1 max = 1, cap = 1 1 1 max = 1, sum(coefficients)=2**1 = 2, cap = 2/1 1 2 1 max = 2, sum = 2**2, cap = 4/2 = 2 units 1 3 3 1 max = 3, sum = 2**3, cap = 8/3 units 1 4 6 4 1 max = 6, sum = 2**4, cap = 16/6 units  #Not sure but this is how it seems to me for the rate @which water is being added.  1 1/2 1/2 1/4 2/4 1/4 1/8 3/8 3/8 1/8 1/16 4/16 6/16 4/16 1/16  Should I use 2-D list and define as : Δ1, Δ2 = 0, 0 if g(n-1, k)>1 and k <= n-1: Δ1 = g(n-1, k) -1 if g(n-1, k-1)>1 and k-1 <= n-1: Δ2 = g(n-1, k-1) - 1 g(n, k) = Δ1/2 + Δ2/2  g(n,k) = g(n-1, k-1) + g(n-1, k) g = [[0]*(i+1) for i in range(11)] def f(g, K): g[1][1] += 1 K = K-1 d1, d2 = 0, 0 for n in range(2, 10): for k in range(1, n+1): if k ==1: g[n][k] = g[n-1][k]/2 if k == n: g[n][k] = g[n-1][k-1]/2 else: if g[n-1][k-1]>1: d1 = g[n-1][k-1] -1 if g[n-1][k] > 1: d2 = g[n-1][k] -1 g[n][k] = d1/2 + d2/2 return g, K k = int(input()) while k: g, k = f(g, k) for x in g: print(x)  I don't know what is missing?

Answer: For such small K constraint simple row-by-row filling is enough (we can store only two rows, here 2D list is used for simplicity) def fillGlasses(k, row, col): gl = [[k]] level = 1 overflow_occured = True while overflow_occured: # also can stop when at needed row print(gl[level-1]) #before overflow level += 1 overflow_occured = False gl.append([0]*level) for i in range(level - 1): t = gl[level-2][i] - 1 if t > 0: gl[level-1][i] += t/2 gl[level-1][i+1] += t/2 gl[level-2][i] = 1 overflow_occured = True #print(gl) #after all return gl[row-1][col-1] print(fillGlasses(21,8,4)) [21] [10.0, 10.0] [4.5, 9.0, 4.5] [1.75, 5.75, 5.75, 1.75] [0.375, 2.75, 4.75, 2.75, 0.375] [0, 0.875, 2.75, 2.75, 0.875, 0] [0, 0, 0.875, 1.75, 0.875, 0, 0] [0, 0, 0, 0.375, 0.375, 0, 0, 0] 0.375