eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In each group, each 1st unique item should be given a different number in new column 'num'. I can form the groups but I don't know how to number the unique pieces. Is there a way to do that ? Unique numbers are: AF=1 / CT=2 / RT=4 / CTS=4 data = {'ATEXT': ['AF', 'AF', '', '', 'CT', 'RT', '', 'AF', 'AF', 'CTS', 'AF', 'AF', 'AF', 'CT', 'AF', 'CT', 'AF', 'AF', 'AF', 'AF', 'RT', 'RT', '', '', 'AF', 'CT', 'CT', 'RT', 'AF', 'AF', 'CT']} df = pd.DataFrame(data) df End Result should be: Out[6]: ATEXT num 0 AF 1 1 AF 2 3 4 CT 2 5 RT 3 6 7 AF 1 8 AF 9 CTS 4 10 AF 11 AF 12 AF 13 CT 2 14 AF 15 CT 16 AF 17 AF 18 AF 19 AF 20 RT 3 21 RT 22 23 24 AF 1 25 CT 2 26 CT 27 RT 3 28 AF 29 AF 30 CT  my idea (does not yet give a useful result) : nl = df['ATEXT']!=("") df['num'] = (df['ATEXT'].mask(nl) .groupby(nl) .where(~df.ATEXT.duplicated() ) 

Answer: IIUC, I using a few intermediate columns to help with logic. Instead of using factorize you could use map to assign your unique numbers. Try: df['CODE'] = df['ATEXT'].mask(df['ATEXT'] == '').factorize()[0] + 1 df.loc[df['ATEXT'] == '', 'CODE'] = np.nan df['grp'] = df['ATEXT'].eq('').cumsum() df['Num'] = df.groupby('grp', group_keys=False).apply(lambda x: x.drop_duplicates('ATEXT'))['CODE'] df['Num'] = df['Num'].fillna('') df[['ATEXT', 'Num']]  Output:  ATEXT Num 0 AF 1.0 1 AF 2 3 4 CT 2.0 5 RT 3.0 6 7 AF 1.0 8 AF 9 CTS 4.0 10 AF 11 AF 12 AF 13 CT 2.0 14 AF 15 CT 16 AF 17 AF 18 AF 19 AF 20 RT 3.0 21 RT 22 23 24 AF 1.0 25 CT 2.0 26 CT 27 RT 3.0 28 AF 29 AF 30 CT  Or, much like you were doing, let use transform with groupby to mask duplicates with where and duplicated in each group: df['Num_2'] = df.groupby('grp')['CODE'].transform(lambda x: x.where(~x.duplicated(keep='first')).fillna('')) df[['ATEXT', 'Num_2']]  Output:  ATEXT Num_2 0 AF 1.0 1 AF 2 3 4 CT 2.0 5 RT 3.0 6 7 AF 1.0 8 AF 9 CTS 4.0 10 AF 11 AF 12 AF 13 CT 2.0 14 AF 15 CT 16 AF 17 AF 18 AF 19 AF 20 RT 3.0 21 RT 22 23 24 AF 1.0 25 CT 2.0 26 CT 27 RT 3.0 28 AF 29 AF 30 CT
Question: I understand that Pydantic can export models to JSON, see ref. But in practice, this means instances of a model: from datetime import datetime from pydantic import BaseModel class BarModel(BaseModel): whatever: int class FooBarModel(BaseModel): foo: datetime bar: BarModel m = FooBarModel(foo=datetime(2032, 6, 1, 12, 13, 14), bar={'whatever': 123}) print(m.json()) #> {"foo": "2032-06-01T12:13:14", "bar": {"whatever": 123}}  My question is parsing the class itself; so that the class could be understood by other languages, for example JavaScript. Here is an example of what I'm envisioning: {BarModel: { 'whatever': 'int'} } {FooBarModel: { 'foo': 'datetime', 'bar': {BarModel: {'whatever': 'int'}} }  Is this possible using built-in Pydantic functionality?

Answer: Pydantic has built-in functionality to generate the JSON Schema of your models. This is a standardised format that other languages will have tooling to deal with. For example, with your definitions, running: import json print(json.dumps(FooBarModel.model_json_schema(), indent=2))  would output: { "$defs": { "BarModel": { "properties": { "whatever": { "title": "Whatever", "type": "integer" } }, "required": [ "whatever" ], "title": "BarModel", "type": "object" } }, "properties": { "foo": { "format": "date-time", "title": "Foo", "type": "string" }, "bar": { "$ref": "#/$defs/BarModel" } }, "required": [ "foo", "bar" ], "title": "FooBarModel", "type": "object" }
Question: Returning an overloaded function from a higher-order callable does not produce expected results with respect to type hints: namely, the resulting function behaves as if it was unannotated at all. Here a minimal example: from typing import Optional, overload def get_f(b: int): @overload def f(x: int) -> str: ... @overload def f(x: int, c: bool) -> int: ... def f(x: int, c: Optional[bool] = None) -> int | str: return 2 * x * b if c is not None else "s" return f d = get_f(2) a = d(2) b = a ** 2  Here mypy does not realize that b = a**2 leads to an TypeError. How to remedy this?

Answer: You can create a type denoting an overloaded callable using typing.Protocol (read the PEP to learn more about structural subtyping): from typing import Optional, Protocol, overload class FProtocol(Protocol): @overload def __call__(self, x: int) -> str: ... @overload def __call__(self, x: int, c: bool) -> int: ... def get_f(b: int) -> FProtocol: @overload def f(x: int) -> str: ... @overload def f(x: int, c: bool) -> int: ... def f(x: int, c: Optional[bool] = None) -> int | str: return 2 * x * b if c is not None else "s" return f d = get_f(2) a = d(2) b = a ** 2 # E: Unsupported operand types for ** ("str" and "int") [operator]  mypy playground pyright playground Using protocol is the only way to specify an overloaded callable, as far as I know. When you leave the return type unspecified, it defaults to Any, as PEP484 mandates:  For a checked function, the default annotation for arguments and for the return type is Any
Question: I have many dataframe like: df1 df2 and so on... gene | counts gene | counts KRAS 136 KRAS 96 DNAH5 3 DNAH5 4 TP53 105 TP53 20  I want to merge them and sum the column 'counts' so I end with only one dataframe merged_df gene | counts KRAS 232 DNAH5 7 TP53 125  I have tried to use pd.merge but it only accepts 2 dataframes at once, I have 14 dataframes. I used pd.concat for multiple dataframes but can't sum them after.

Answer: Indeed, pd.merge only merges 2 dataframes. But pd.join can join many, if they have the same index: # Some example data. Note the None in `df3`. We want our code to handle that well. df1 = pd.DataFrame({'gene': ['KRAS', 'DNAH5', 'TP53'], 'counts': [136, 3, 105]}) df2 = pd.DataFrame({'gene': ['KRAS', 'DNAH5', 'TP53'], 'counts': [96, 4, 20]}) df3 = pd.DataFrame({'gene': ['KRAS', 'DNAH5', 'TP53'], 'counts': [1000, None, 3000]}) dfs = [df1, df2, df3] # We need the same index for the pd.DataFrame.join to work dfs = [df.set_index('gene') for df in dfs] # All the non-index columns need unique names, so chanding `columns` to `columns_0`, `columns_1` dfs = [df.rename(columns={'counts': f'counts_{i}'}) for i, df in enumerate(dfs)] # Actual join. We are joining first with the rest df = dfs[0].join(dfs[1:], how='outer') # Since we don't have any other data, we can just sum all columns. df.sum(1)  This prints: gene KRAS 1232.0 DNAH5 7.0 TP53 3125.0
Question: I'm trying to read multiple csv.gz files into a dataframe but it's not working as I expect. When I use this globbing pattern: pl.read_csv('folder_1\*.csv.gz') It returns this error:  ComputeError: cannot scan compressed csv; use read_csv for compressed data   This error occurred with the following context stack: >[1] 'csv scan' failed [2] 'select' input failed to resolve  Which is strange considering I'm using the very function they suggest. However, passing this globbing pattern for csv works completely fine: pl.read_csv('folder_1\*.csv') How can I get around this? I'm currently just using glob.glob() and iterating through the list but I thought it'll look neater without it.

Answer: When I pass a glob string blah/blah/blah/*.csv.gz to pl.read_csv, it passes this to pl.scan_csv because it is a glob string. See polars.io.csv.functions line 514 et seq in version 1.1.0. There are two separate questions here:  How do you read multiple CSVs? You can read multiple CSVs by passing a glob string to pl.scan_csv. It returns a lazy data frame that you can then evaluate with .collect().  How do you read a compressed CSV? You can read certain types of compressed CSVs with pl.read_csv (notable exception being csv.xzs which do not work).   But put the two questions together and it turns out pl.scan_csv does not support compressed files at all. This is an open issue. If you want a one liner for reading your CSVs, you will have to fall back on something like a list comprehension with eager execution: from glob import glob l = [pl.read_csv(i) for i in glob('*.csv.gz')]  Then do what you will with the list of CSVs (eg pl.concat).
Question: I was asked the following in an interview:  Given a sorted array with n numbers (where n is a multiple of 4), return whether any number appears more than n/4 times.  My initial thought was to iterate over the array and keep a counter: limit = len(nums) // 4 counter = 1 for i in range(1, len(nums)): if nums[i] == nums[i-1]: counter += 1 if counter > limit: return True else: counter = 1 return False  However the interviewer asked if I could do better. Sorted array, better than linear complexity, I immediately thought "binary search"... I explored this path but was clueless. After a few minutes, he gave a hint: "if any i exists where nums[i] == nums[i + len(nums) / 4]", does that mean we should return true? I was trying to apply this to my binary search and was not thinking straight so the hint went over my head... Retrospectively it is trivial, however I only see how to apply this with a sliding window: limit = len(nums) // 4 for i in range(limit, len(nums)): if nums[i] == nums[i-limit]: return True return False  Is it what the interviewer meant? Or is there a better, non-linear solution?

Answer: A number that appears more than n/4 times in nums must be one of the elements of nums[::len(nums)//4]. For each candidate number, we perform a binary search for the first position where it appears, and check whether the same number appears len(nums)//4 spots later: import bisect def check(nums): n = len(nums) assert n % 4 == 0 for i in nums[::n//4]: first_appearance = bisect.bisect_left(nums, i) if first_appearance + n//4 < len(nums) and nums[first_appearance + n//4] == i: return True return False  We perform at most 4 binary searches, and at most a constant amount of other work, so the time complexity is O(log n).
Question: I want to use a given colormap (let's say viridis) and create plots and colorbars with discrete colors from that colormap. I used to use mpl.cm.get_cmap("viridis", 7) for 7 different colors, but this function is deprecated and will be removed. The recommendation is to use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap(obj) instead, but neither of these allow to specify a number of discrete colors. So far I have only found complicated workarounds online, is anyone aware of a simple, straightforward way as I had originally? Thank you! Here is a sample code with simplified data. It took me a while to get the colorbar axis right how I wanted it, so I would prefer to stick with what I already have instead of a different plt.colorbar() solution. import os.path as op import numpy as np import matplotlib as mpl import matplotlib.pyplot as plt mpl.rcParams.update({'font.size': 30}) from mpl_toolkits.axes_grid1 import make_axes_locatable PLOT = '/tmp/' def main(): data = np.random.random((20,20)) data[5,:] = np.nan fig, ax = plt.subplots(figsize=(8.3,12)) divider = make_axes_locatable(ax) cm = mpl.cm.get_cmap('viridis', 7) cm.set_bad('darkgrey', alpha=1) plt.pcolormesh(data, cmap=cm, vmin=0,vmax=1) ax.axis('off') cax = divider.append_axes("right", size="5%", pad=0.2) cb = plt.colorbar(cax=cax) plt.savefig(op.join(PLOT, 'test.png'), bbox_inches='tight', dpi=300) plt.clf() plt.close() if __name__ == "__main__": main() 

Answer: A simple option to replace matplotlib.cm.get_cmap (deprecated in 3.7.0) is to use matplotlib.pyplot.get_cmap (that was preserved in the API for backward compatibility). Another one woud be to make a resampled colormap (used under the hood) :  resampled(lutsize) [source] Return a new colormap with lutsize entries.  Note that the lut is a parameter in get_cmap(name=None, lut=None).  cm = mpl.cm.get_cmap("viridis", 7) # (-) cm = plt.get_cmap("viridis", 7) # (+)
Question: If one add float32 and float64 scalars, the result is float64: float32 is promoted to float64. However, I find that, adding a float32 array and a float64 scalar, the result is a float32 array, rather than one might expect to be a float64 array. I have written some code repreduce the problem. My question is why the dtype of np.add(np.array([a]), b) is float32? import numpy as np import sys print(sys.version_info) # sys.version_info(major=3, minor=10, micro=9, releaselevel='final', serial=0) print(np.__version__) # '1.24.1' a = np.float32(1.1) b = np.float64(2.2) print((a+b).dtype) # float64 print(np.add(a, np.array([b])).dtype) # float64 print(np.add(np.array([a]), np.array([b])).dtype) # float64 print(np.add(np.array([a]), b).dtype) # float32 ?? expcting float64  This contradicts the doc of np.add (https://numpy.org/doc/stable/reference/generated/numpy.add.html), which says in the Notes  Equivalent to x1 + x2 in terms of array broadcasting.  x, y = numpy.broadcast_arrays(np.array([a]), b) print(np.add(x, y).dtype) ## float64 

Answer:  why the dtype of np.add(np.array([a]), b) is float32?  Because NumPy's promotion rules involving scalars pre-NEP 50 were confusing*. Using the NEP 50 rules in your version of NumPy, the behavior is as you expect: import numpy as np import sys # Use this to turn on the NEP 50 promotion rules np._set_promotion_state("weak") print(sys.version_info) # sys.version_info(major=3, minor=10, micro=9, releaselevel='final', serial=0) print(np.__version__) # '1.24.0' a = np.float32(1.1) b = np.float64(2.2) print((a+b).dtype) # float64 print(np.add(a, np.array([b])).dtype) # float64 print(np.add(np.array([a]), np.array([b])).dtype) # float64 print(np.add(np.array([a]), b).dtype) # float64  These rules are the default in NumPy 2.0. *: More specifically, the old rules depend on the values. If you set a = b = np.finfo(np.float32).max, the result would overflow if kept in float32, so you would get float64 even with the old rules.
Question: I have two tables df1 = pd.DataFrame([{'a': 1}, {'a': 2}, {'a': 8}]) df1['b'] = "" df2 = pd.DataFrame([{'e': [1,2,3], 'f': 1},{'e': [4,5,6], 'f': 2},{'e': [7,8,9], 'f': 3}]) e f  I would like to insert a value into df1['b'] from df2['f'] based on the condition that value in df['a'] is somewhere in df2['e'] Mostly I would use apply/map df1['a'].apply(lambda x: function) I can get the specific value I need df2['f'].loc[df2['e'].apply(lambda y: value_to_look_for in y)].item() but when I put it together df1['a'].apply(lambda x: (df2['f'].loc[df2['e'].map(lambda y:x in y)].item())) I'm getting an error "ValueError: can only convert an array of size 1 to a Python scalar" Can I ask for a solution, what am I missing?

Answer: I would explode, create a mapping Series, and map: df1['b'] = df1['a'].map(df2.explode('e').set_index('e')['f'])  Output:  a b 0 1 1 1 2 1 2 8 3  Intermediate mapping Series: df2.explode('e').set_index('e')['f'] e 1 1 2 1 3 1 4 2 5 2 6 2 7 3 8 3 9 3 Name: f, dtype: int64
Question: Here is code I am trying to optimize: import numpy as np rng = np.random.default_rng() num = np.fromiter((rng.choice([0, 1], size=n, p=[1-qEff, qEff]).sum() for n in num), dtype='int')  What it does is it takes array num, where each element represents a bucket with given number of items. For each item it rolls a dice (Bernoulli trial with probability qEff) and if successful, it keeps the item in the bucket, if not it removes it. To avoid the loop, I came with idea to do all the rolling first and then sum number of trials corresponding to particular element in num array. So it would go something like this: rol = rng.choice([0, 1], size=num.sum(), p=[1-qEff, qEff] num = some_smart_sum(rol, num)  So now I need to figure out how to do the some_smart_sum() without a loop. So far it looks like np.add.reduceat() should do the trick, but I am having difficulties building the indices parameter.

Answer: To add buckets you can get the indices for reduceat from a call to cumsum, but you will have to add a 0 in front and get rid of the last entry, which is equal to the total number of items and is added automatically by reduceat. Something like this should work: indices = np.concatenate(([0], np.cumsum(num)[:-1])) num = np.add.reduceat(roll, indices)  A less obvious approach, but probably faster for large inputs is to not do the sums at all... Your sums are the number of successes of n independent Bernoulli trials with success probability p, so they follow a binomial distribution. So you can do that directly instead: rng = np.random.default_rng() num = rng.binomial(num, qEff)
Question: I have below code df = pd.DataFrame(dict( age=[5, 6, np.nan], born=[pd.NaT, pd.Timestamp('1939-05-27'), pd.Timestamp('1940-04-25')], name=['Alfred', 'Batman', np.nan], toy=[np.nan, 'Batmobile', 'Joker']))  Now I want to drop those rows where columns name OR toy have NaN/empty string values. I tried with below code df[~df[['name', 'toy']].isna()]  I was expecting only 2nd row would return. Could you please help where I went wrong?

Answer: what is wrong df[~df[['name', 'toy']].isna()] returns a DataFrame (2D), thus you will apply a mask on the whole DataFrame, keeping values that are truthy (the shape remains unchanged). You're essentially doing: df.where(~df[['name', 'toy']].isna())  Which will mask the non target columns with NaN and the NaN values from name/toy with NaN (= no change) how to fix it What you want to do is boolean indexing, you must pass a Series (and thus aggregate with any). Also, since '' is not considered a NaN, you first have to replace them to None: out = df[~df[['name', 'toy']].replace('', None).isna().any(axis=1)]  Note that this is equivalent to using notna+all without the ~: out = df[df[['name', 'toy']].replace('', None).notna().all(axis=1)]  Output:  age born name toy 1 6.0 1939-05-27 Batman Batmobile
Question: Context:  I am developing a standalone Django package that is intended to be added to existing Django projects. This package includes Django models with methods that interact directly with a database. I want to write tests for these methods using pytest and pytest-django. I need to run these tests in a CI/CD environment, specifically using GitHub Actions.  Problems Encountered:  Database Setup:  Since this is a standalone package, it does not include a manage.py file. This makes it challenging to run typical Django management commands like migrate to set up the database schema for testing.   Table Creation:  I encounter errors indicating that the table for my model does not exist because the necessary migrations are not applied. This is problematic because my tests need to interact with the database tables directly.   Fixtures and Setup:  I need to ensure the database tables are created and populated with initial data before the tests run. My attempts to use pytest fixtures and manually create database tables within the tests have not been successful.    What I've Tried:  Using pytest fixtures to set up the initial database state. Manually creating database tables within the test setup using direct SQL execution. Using the mixer library to populate the database with test data.  Desired Outcome:  A reliable method to ensure the necessary database tables for my models are created before running the tests. Proper setup and teardown of test data within the standalone package. Integration with CI/CD pipelines like GitHub Actions without relying on manage.py commands.  Additional Information:  I am using an in-memory SQLite database for faster test execution. The test environment should be isolated and not rely on external databases or services.  I am looking for guidance on how to approach this problem or best practices for testing Django models in a standalone package using pytest and pytest-django. Any advice or solutions would be greatly appreciated.

Answer: You don't need manage.py to run migrate. The same commands are available via the django-admin command that Django installs, but you'll need to set the DJANGO_SETTINGS_MODULE variable that manage.py would ordinarily set. IOW, if you'd generally run python manage.py and your project settings are myproject.settings, you can just as well run env DJANGO_SETTINGS_MODULE=myproject.settings django-admin migrate. That said, that's not your issue here – pytest-django will deal with setting up and tearing down a testing database for you. Generally, when developing a Django package, you'd have a testing project and app next to it to host the package for development use. An example (of a standalone package, tested with GitHub Actions) is here (repository maintained by yours truly); lippukala is the reusable package, and lippukala_test_app is the package that hosts the settings for pytest-django to use; there's a stanza [tool.pytest.ini_options] DJANGO_SETTINGS_MODULE = "lippukala_test_app.settings"  in the pyproject.toml file that makes that work.
Question: For a list of percentage data, I need to check if the last value (90.2) is somehow higher and somewhat "abnormal" than the rest of the data. Clearly it is in this sequence. delivery_pct = [59.45, 55.2, 54.16, 66.57, 68.62, 64.19, 60.57, 44.12, 71.52, 90.2] But for the below sequnece the last value is not so: delivery_pct = [ 63.6, 62.64, 60.36, 72.8, 70.86, 40.51, 52.06, 61.47, 51.55, 74.03 ] How do I check if the last value is abnormally higher than the rest? About Data: The data point has the range between 0-100%. But since this is percentage of delivery taken for a stock for last 10 days, so it is usually range bound based on nature of stock (highly traded vs less frequently traded), unless something good happens about the stock and there is higher delivery of that stock on that day in anticipation of good news.

Answer: Once you've determined a threshold (deviation from mean) you could do this: import statistics t = 2 # this is the crucial value pct = [59.45, 55.2, 54.16, 66.57, 68.62, 64.19, 60.57, 44.12, 71.52, 90.2] mean = statistics.mean(pct) tsd = statistics.pstdev(pct) * t lo = mean - tsd hi = mean + tsd print(*[x for x in pct if x < lo or x > hi], sep="\n")  Output: 90.2  It's the threshold value that (effectively) determines what's "abnormal" The interquartile range (IQR) method produces the same result: import statistics pct = [59.45, 55.2, 54.16, 66.57, 68.62, 64.19, 60.57, 44.12, 71.52, 90.2] spct = sorted(pct) m = len(spct) // 2 Q1 = statistics.median(spct[:m]) m += len(spct) % 2 # increment m if list length is odd Q3 = statistics.median(spct[m:]) IQR = Q3 - Q1 lo = Q1 - 1.5 * IQR hi = Q3 + 1.5 * IQR print(*[x for x in pct if x < lo or x > hi], sep="\n")  Output: 90.2
Question: I'm using pyenv to manage my python versions. When i use python 3.12.4 or python3.9^, i get this error:  File "src/pymssql/_pymssql.pyx", line 1, in init pymssql._pymssql ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pymssql/_mssql.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'  my script: from sqlalchemy import create_engine, text import bi_pass v_server = bi_pass.v_server v_user = bi_pass.v_user v_password = bi_pass.v_password v_database = bi_pass.v_database source_engine = create_engine(f"mssql+pymssql://{v_user}:{v_password}@{v_server}:1433/{v_database}") source_conn = source_engine.connect() query = text( """ SELECT * from test""") result = source_conn.execute(query) rows = result.fetchall() source_conn.close() source_engine.dispose()  I have a Mac m3 chip, how can I fix it? Thanks

Answer: also another way that works: pip install --no-binary :all: pymssql --no-cache --force