eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: # hierarchical indices and columns index = pd.MultiIndex.from_product([[2013, 2014], [1, 2]], names=['year', 'visit']) columns = pd.MultiIndex.from_product([['Bob', 'Guido', 'Sue'], ['HR', 'Temp']], names=['subject', 'type']) # mock some data data = np.round(np.random.randn(4, 6), 1) data[:, ::2] *= 10 data += 37 # create the DataFrame health_data = pd.DataFrame(data, index=index, columns=columns) health_data  How can I get all rows that are using pd.query:  Bob HR > 35 any subjects HR > 35  If this is not possible with pd.query at the moment, this would be the accepted answer.

Answer: You can't use MultiIndexes with query. (Well, you can to some limit use health_data.query('@health_data.Bob.HR > 35'), but this has many flaws, and you won't be able to do this with a sublevel only). You can use xs instead to select your levels: # is Bod/HR > 35? m1 = health_data[('Bob', 'HR')].gt(35) # is any HR > 35? m2 = health_data.xs('HR', level='type', axis=1).gt(35).any(axis=1) # keep rows with both conditions out = health_data.loc[m1&m2]  output: subject Bob Guido Sue type HR Temp HR Temp HR Temp year visit 2013 1 42.0 38.0 45.0 37.7 33.0 36.9 2 48.0 38.2 42.0 36.2 21.0 35.9 2014 2 50.0 39.2 44.0 36.0 36.0 35.8
Question: Python 3.11 introduces new two parameters to the dis.dis function, show_caches and adaptive.  >>> import dis >>> >>> help(dis.dis) Help on function dis in module dis: dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False) Disassemble classes, methods, functions, and other compiled objects. With no argument, disassemble the last traceback. Compiled objects currently include generator objects, async generator objects, and coroutine objects, all of which store their code object in a special attribute.  What does this parameters means in python 3.11?. I did check the result by setting it to True but the result remains same as like setting it to False. >>> dis.dis("a = 1", show_caches=True, adaptive=True) 0 0 RESUME 0 1 2 LOAD_CONST 0 (1) 4 STORE_NAME 0 (a) 6 LOAD_CONST 1 (None) 8 RETURN_VALUE >>> >>> >>> dis.dis("a = 1") 0 0 RESUME 0 1 2 LOAD_CONST 0 (1) 4 STORE_NAME 0 (a) 6 LOAD_CONST 1 (None) 8 RETURN_VALUE 

Answer: The show_caches argument is described at the very top of dis documentation since it is possible to pass it to many of the module's functions:  Changed in version 3.11: Some instructions are accompanied by one or more inline cache entries, which take the form of CACHE instructions. These instructions are hidden by default, but can be shown by passing show_caches=True to any dis utility.  adaptive is described under CACHE documentation:  Rather than being an actual instruction, this opcode is used to mark extra space for the interpreter to cache useful data directly in the bytecode itself. It is automatically hidden by all dis utilities, but can be viewed with show_caches=True. Logically, this space is part of the preceding instruction. Many opcodes expect to be followed by an exact number of caches, and will instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data. New in version 3.11.
Question: I have quite a complex If statement that I would like to add as a column in my pandas dataframe. In the past I've always used numpy.select for this type of problem, however I wouldn't know how to achieve that with a multi-line if statement. I was able to get this in Excel: =IF(sum1=3,IF(AND(col1=col2,col2=col3),0,1),IF(sum1=2,IF(OR(col1=col2,col2=col3,col1=col3),0,1),IF(sum1=1,0,1)))  and write it in Python just as a regular multi-line 'if statement', just want to find out if there is a far cleaner way of presenting this. if df['sum1'] == 3: if df['col1'] == df['col2'] and df['col2'] == df['col3']: df['verify_col'] = 0 else: df['verify_col'] = 1 elif df['sum1'] == 2: if df['col1'] == df['col2'] or df['col2'] == df['col3'] or df['col1'] == df['col3']: df['verify_col'] = 0 else: df['verify_col'] = 1 elif df['sum1'] == 1: df['verify_col'] = 0 else: df['verify_col'] = 1  Here's some sample data: df = pd.DataFrame({ 'col1': ['BMW', 'Mercedes Benz', 'Lamborghini', 'Ferrari', null], 'col2': ['BMW', 'Mercedes Benz', null, null, 'Tesla'], 'col3': ['BMW', 'Mercedes', 'Lamborghini', null, 'Tesla_'], 'sum1': [3, 3, 2, 1, 2] })  I want a column which has the following results: 'verify_col': [0, 1, 0, 0, 1]  It basically checks whether the columns match for those that have values in them and assigns a 1 or a 0 for each row. 1 meaning they are different, 0 meaning zero difference.

Answer: Use numpy.where with chain mask with | for bitwise OR - if no match any conditions is created 1: m1 = (df['sum1'] == 3) m2 = (df['col1'] == df['col2']) & (df['col2'] == df['col3']) m3 = (df['sum1'] == 2) m4 = (df['col1'] == df['col2']) | (df['col2'] == df['col3']) | (df['col1'] == df['col3']) m5 = df['sum1'] == 1 df['verify_col'] = np.where((m1 & m2) | (m3 & m4) | m5, 0, 1)  If need None if no match any conditions: df['verify_col'] = np.select([(m1 & m2) | (m3 & m4) | m5, (m1 & ~m2) | (m3 & ~m4) | ~m5], [0,1], default=None) print (df) col1 col2 col3 sum1 verify_col 0 BMW BMW BMW 3 0 1 Mercedes Benz Mercedes Benz Mercedes 3 1 2 Lamborghini NaN Lamborghini 2 0 3 Ferrari NaN NaN 1 0 4 NaN Tesla Tesla_ 2 1
Question: I currently have an apache-beam pipeline in Python in which I'm reading parquet, converting it to a dataframe to do some pandas cleaning, and then converting back to parquet where I'd like to then write the file. It looks like this:  with beam.Pipeline(options=pipeline_options) as p: dataframes = p \ | 'Read' >> beam.io.ReadFromParquetBatched(known_args.input) \ | 'Convert to pandas' >> beam.Map(lambda table: table.to_pandas()) \ | 'Process df' >> beam.ParDo(ProcessDataFrame()) \ | 'Convert to parquet' >> beam.Map(lambda table: table.to_parquet()) \ | 'Write to parquet' >> beam.io.WriteToParquet(known_args.output)  However this returns an error as expected because I'm missing the schema as an argument to WriteToParquet. Traceback (most recent call last): File "/Users/kgallatin/dataflow/example.py", line 75, in <module> main() File "/Users/kgallatin/dataflow/example.py", line 70, in main | 'Write to parquet' >> beam.io.WriteToParquet(known_args.output) TypeError: __init__() missing 1 required positional argument: 'schema'  I have 100-1000s of columns that potentially will change over the lifetime of this pipeline, so I'd like to avoid manually writing them all in the pyarrow format as described here. When I print the parquet from the previous step I can see a pandas schema and some binary pyarrow stuff - is there a way to extract the schema from the parquet at this step for use?

Answer: Your codetable.to_parquet() returns serialized bytes of all of the records in Parquet format. Currently, WriteToParquet() need a manually given fixed schema. So, you have to implement your own pipeline(PTransform) if you want to generate a schema automatically. If you don't need a parallelism(such as sharding), you can do like the following using the low level API. ... import pyarrow as pa from apache_beam.io.filesystems import FileSystems from apache_beam.io.filesystem import CompressionTypes ... def write_dataframe_to_parquet(frame): table = pa.Table.from_pandas(frame) with FileSystems.create(known_args.output, mime_type='application/x-parquet', compression_type=CompressionTypes.UNCOMPRESSED, ) as f: pa.parquet.write_table(table, f) ... dataframes = p \ ... | 'Process df' >> beam.ParDo(ProcessDataFrame()) \ | 'Write to parquet' >> beam.Map(write_dataframe_to_parquet)
Question: I've got an array with data like this a = [[1,2,3],[4,5,6],[7,8,9]]  and I want to change it to b = [[1,1,2,2,3,3],[1,1,2,2,3,3],[4,4,5,5,6,6],[4,4,5,5,6,6],[7,7,8,8,9,9],[7,7,8,8,9,9]]  I've tried to use numpy.resize() function but after resizing, it gives [[1,2,3,4],[1,2,3,4],[1,2,3,4],[1,2,3,4]]. I can use a for loop to put the numbers at the indexes I need but just wondering if there is any easier way of doing that? To visualise the task, here is the original array  This is what I want 

Answer: My initial though was that np.tile would work but in fact what you are looking for is np.repeat twice on two different axes. Try this runnable example!   #!/usr/bin/env python import numpy as np a = [[1,2,3],[4,5,6],[7,8,9]] b = np.repeat(np.repeat(a, 2, axis=1), 2, axis=0) b <script src="https://modularizer.github.io/pyprez/pyprez.min.js"></script>
Question: Have two dataframes, and I am attempting to 'merge' with conditions. The idea being if df_B's DateTime is greater than ST's DateTime by less than 5 mins, then add the value of df_B 'MC' into a new column in df_A. If not, its a blank.  df_A = {'SS': ['2022/10/23 9:58:08', '2022/10/23 9:58:08', '2022/10/23 23:07:17', '2022/10/23 23:07:17', '2022/10/24 2:06:48', '2022/10/24 2:06:48', '2022/10/24 5:31:44', '2022/10/24 5:31:44'], 'SC': [6764, 6764, 6778, 6778, 6782, 6782, 6787, 6787] } df_B = {'DateTime': ['2022/10/23 6:05:51', '2022/10/23 6:06:51', '2022/10/23 7:20:51', '2022/10/23 7:21:51', '2022/10/23 7:51:51', '2022/10/23 7:52:51', '2022/10/24 5:31:58', '2022/10/24 5:35:58'], 'MC': [871.71, 871.77, 871.78, 871.77, 871.78, 871.77, 866.90, 866.91] } df_A = pd.DataFrame(df_A) df_A['SS'] = pd.to_datetime(df_A['SS']) df_B = pd.DataFrame(df_B) df_B['DateTime'] = pd.to_datetime(df_B['DateTime'])  Desired Result I have mostly tried pd.merge_asof() and converting the datatimes to indexes and then left/right_index True and direction 'forward'. df = pd.merge_asof(left=df_A, right=df_B, left_index=True, right_index=True, direction='forward')  The problem is the merge inserts an MC values in every row based on the direction. And very rarely (or never) id get an exact datetime match). Kinda given up using merge. Should I just parse df_A datetime, check if df_B's datetime its greater than df_A datetime (but reject if greater than 5 mins), and then find the corresponding MC value? For this method, I would normally use the lambda function with apply.  df_A.apply(lambda x: (x['SS'], x['SC']), axis=1)  So how to incorporate df_B into the above, or should call a function with the method, the function do the processing? Hope makes sense. tks

Answer: I get that the rows of df_A and df_B are aligned and both dataframes have equal number of rows. In this case:  Putting the 'on' columns into indexes is not necessary to merge dataframes. Instead you can use 'left_on' and 'right_on'. Use 'tolerance' along with the 'forward' direction for the required condition. Reset indexes into "index" columns in order to align the rows by "index" before the merge.  BTW, make sure before the merge that  Both DataFrames must be sorted by the key.  https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html#pandas.merge_asof df_A.reset_index(inplace=True) df_B.reset_index(inplace=True) df = pd.merge_asof(df_A, df_B, left_on="SS", right_on="DateTime", by="index", direction="forward", tolerance=pd.Timedelta("5m")) index SS SC DateTime MC 0 0 2022-10-23 09:58:08 6764 NaT NaN 1 1 2022-10-23 09:58:08 6764 NaT NaN 2 2 2022-10-23 23:07:17 6778 NaT NaN 3 3 2022-10-23 23:07:17 6778 NaT NaN 4 4 2022-10-24 02:06:48 6782 NaT NaN 5 5 2022-10-24 02:06:48 6782 NaT NaN 6 6 2022-10-24 05:31:44 6787 2022-10-24 05:31:58 866.90 7 7 2022-10-24 05:31:44 6787 2022-10-24 05:35:58 866.91
Question: I generated a Pydantic model and would like to import it into SQLModel. Since said model does not inherit from the SQLModel class, it is not registered in the metadata which is why SQLModel.metadata.create_all(engine)  just ignores it. In this discussion I found a way to manually add models: SQLModel.metadata.tables["hero"].create(engine)  But doing so throws a KeyError for me. SQLModel.metadata.tables["sopro"].create(engine) KeyError: 'sopro'  My motivation for tackling the problem this way is that I want to generate an SQLModel from a simple dictionary like this: model_dict = {"feature_a": int, "feature_b": str}  And in this SO answer, I found a working approach. Thank you very much in advance for your help!

Answer: As far as I know, it is not possible to simply convert an existing Pydantic model to an SQLModel at runtime. (At least as of now.) There are a lot of things that happen during model definition. There is a custom meta class involved, so there is no way that you can simply substitute a regular Pydantic model class for a real SQLModel class, short of manually monkeypatching all the missing pieces. That being said, you clarified that your actual motivation was to be able to dynamically create an SQLModel class at runtime from a dictionary of field definitions. Luckily, this is in fact possible. All you need to do is utilize the Pydantic create_model function and pass the correct __base__ and __cls_kwargs__ arguments: from pydantic import create_model from sqlmodel import SQLModel field_definitions = { # your field definitions here } Hero = create_model( "Hero", __base__=SQLModel, __cls_kwargs__={"table": True}, **field_definitions, )  With that, SQLModel.metadata.create_all(engine) should create the corresponding database table according to your field definitions. See this question for more details. Be sure to use correct form for the field definitions, as the example you gave would not be valid. As the documentation says, you need to define fields in the form of 2-tuples (or just a default value): model_dict = { "feature_a": (int, ...), "feature_b": (str, ...), "feature_c": 3.14, }  Hope this helps.
Question: On Databricks, it is possible to install Python packages directly from a git repo, or from the dbfs: %pip install git+https://github/myrepo %pip install /dbfs/my-library-0.0.0-py3-none-any.whl  Is there a way to enable a live package development mode, similar to the usage of pip install -e, such that the databricks notebook references the library files as is, and it's possible to update the library files on the go? E.g. something like %pip install /dbfs/my-library/ -e  combined with a way to keep my-library up-to-date? Thanks!

Answer: I would recommend to adopt the Databricks Repos functionality that allows to import Python code into a notebook as a normal package, including the automatic reload of the code when Python package code changes. You need to add the following two lines to your notebook that uses the Python package that you're developing: %load_ext autoreload %autoreload 2  Your library is recognized as the Databricks Repos main folders are automatically added to sys.path. If your library is in a Repo subfolder, you can add it via: import os, sys sys.path.append(os.path.abspath('/Workspace/Repos/<username>/path/to/your/library'))  This works for the notebook node, however not for worker nodes. P.S. You can see examples in this Databricks cookbook and in this repository.
Question: I am trying to read data from excel and store the text data in firestore. When I try to add it one by one without for loop it is working but if I try to automate the process it is not working. Working Code: import firebase_admin from firebase_admin import credentials from firebase_admin import firestore cred = credentials.Certificate("certificate.json") firebase_admin.initialize_app(cred) db=firestore.client() name = 'Ashwin' date = '23/10/2022' roll = 'sampledata' cert = 'sampledata' def store_add(name, date,roll,cert): data = {'name':name, 'date':date,'roll':roll} db.collection('certs').document(cert).set(data) store_add(name,date,roll,cert)  This is my code (Not Working): import pandas as pd import firebase_admin from firebase_admin import credentials from firebase_admin import firestore cred = credentials.Certificate("certificate.json") firebase_admin.initialize_app(cred) db=firestore.client() def store_add(name, date,roll,cert): data = {'name':name, 'date':date,'roll':roll} db.collection('certs').document(cert).set(data) df = pd.read_excel('data.xlsx') name_list = list(df['name']) cert = list(df['cert']) roll = list(df['roll']) date="24/10/2022" for i in range(len(name_list)): store_add(name_list[i],date,roll[i],cert[i]) print("Added:",name_list[i])  I am getting the following error: Traceback (most recent call last): File "e:\PSDC\Certificate-Generator\bulk.py", line 23, in <module> store_add(name_list[i],date,roll[i],cert[i]) File "e:\PSDC\Certificate-Generator\bulk.py", line 14, in store_add db.collection('certs').document(cert).set(data) File "C:\Users\ashwi\AppData\Local\Programs\Python\Python310\lib\site-packages\google\cloud\firestore_v1\base_collection.py", line 130, in document return self._client.document(*child_path) _init__ super(DocumentReference, self).__init__(*path, **kwargs) File "C:\Users\ashwi\AppData\Local\Programs\Python\Python310\lib\site-packages\google\cloud\firestore_v1\base_document.py", line 60, in __init__ _helpers.verify_path(path, is_collection=False) File "C:\Users\ashwi\AppData\Local\Programs\Python\Python310\lib\site-packages\google\cloud\firestore_v1\_helpers.py", line 150, in verify_path raise ValueError("A document must have an even number of path elements") ValueError: A document must have an even number of path elements  Firestore Database Structure (Image)

Answer: As mentioned in the documentation, document IDs cannot contain a slash but in the provided screenshot there are 3. The value of cert is PSDC/2022/TCS/01 so the final path becomes certs/PSDC/2022/TCS/01 that has 5 segments i.e. a sub-collection. You can either replace the / with some other character like _: db.collection('certs').document(cert.replace("/", "_")).set(data)  Alternatively, if the slashes are required, you can store the cert ID in a field in document data and use a random document ID.
Question: What is the most pythonic way to reverse the elements of one list based on another equally-sized list? lista = [1,2,4] listb = ['yes', 'no', 'yep'] # expecting [-1, 2,-4] [[-x if y in ['yes','yep'] else x for x in lista] for y in listb] # yields [[-1, -2, -4], [1, 2, 4], [-1, -2, -4]]  if listb[i] is yes or yep, result[i] should be the opposite of lista. Maybe a lambda function applied in list comprehension?

Answer: using zip? >>> [-a if b in ('yes','yep') else a for a,b in zip(lista, listb)] [-1, 2, -4]
Question: When I run the spacy lemmatizer, it does not lemmatize the word "consulting" and therefore I suspect it is failing. Here is my code: nlp = spacy.load('en_core_web_trf', disable=['parser', 'ner']) lemmatizer = nlp.get_pipe('lemmatizer') doc = nlp('consulting') print([token.lemma_ for token in doc])  And my output: ['consulting'] 

Answer: The spaCy lemmatizer is not failing, it's performing as expected. Lemmatization depends heavily on the Part of Speech (PoS) tag assigned to the token, and PoS tagger models are trained on sentences/documents, not single tokens (words). For example, parts-of-speech.info which is based on the Stanford PoS tagger, does not allow you to enter single words. In your case, the single word "consulting" is being tagged as a noun, and the spaCy model you are using deems "consulting" to be the appropriate lemma for this case. You'll see if you change your string instead to "consulting tomorrow", spaCy will lemmatize "consulting" to "consult" as it is tagged as a verb (see output from the code below). In short, I recommend not trying to perform lemmatization on single tokens, instead, use the model on sentences/documents as it was intended. As a side note: make sure you understand the difference between a lemma and a stem. Read this section provided on Wikipedia Lemma (morphology) page if you are unsure. import spacy nlp = spacy.load('en_core_web_trf', disable=['parser', 'ner']) doc = nlp('consulting') print([[token.pos_, token.lemma_] for token in doc]) # Output: [['NOUN', 'consulting']] doc_verb = nlp('Consulting tomorrow') print([[token.pos_, token.lemma_] for token in doc_verb]) # Output: [['VERB', 'consult'], ['NOUN', 'tomorrow']]  If you really need to lemmatize single words, the second approach on this GeeksforGeeks Python lemmatization tutorial produces the lemma "consult". I've created a condensed version of it here for future reference in case the link becomes invalid. I haven't tested it on other single tokens (words) so it may not work for all cases. # Condensed version of approach #2 given in the GeeksforGeeks lemmatizer tutorial: # https://www.geeksforgeeks.org/python-lemmatization-approaches-with-examples/ import nltk from nltk.stem import WordNetLemmatizer nltk.download('averaged_perceptron_tagger') from nltk.corpus import wordnet # POS_TAGGER_FUNCTION : TYPE 1 def pos_tagger(nltk_tag): if nltk_tag.startswith('J'): return wordnet.ADJ elif nltk_tag.startswith('V'): return wordnet.VERB elif nltk_tag.startswith('N'): return wordnet.NOUN elif nltk_tag.startswith('R'): return wordnet.ADV else: return None lemmatizer = WordNetLemmatizer() sentence = 'consulting' pos_tagged = nltk.pos_tag(nltk.word_tokenize(sentence)) lemmatized_sentence = [] for word, tag in pos_tagged: lemmatized_sentence.append(lemmatizer.lemmatize(word, pos_tagger(tag))) print(lemmatized_sentence) # Output: ['consult']
Question: I have the following data frame:  df_test = pd.DataFrame({"f":['a', 'a', 'a', 'a', 'b', 'b', 'b', 'b'], "d":['x', 'x', 'y', 'y', 'x', 'x', 'y', 'y'], "low": [0,5,2,4,5,10,4,8], "up": [5,10,4,6,10,15,8,12], "z": [1,3,6,2,3,7,5,10]})  and what I first have to do is to convert the columns 'low', 'up' and 'z' to list for each (grouped by) 'f' and 'd'. so this is what I did:  dff = df_test.groupby(['f','d'])[['low', 'up', 'z']].agg(list).reset_index()  and this is what I get:  Now I want to extract the last value from the lists in column 'up' and add it to the lists in column 'low'. But this is unfortunately not working:  dff['last'] = (dff['up'].apply(lambda x: x[-1])).tolist() dff['new'] = dff['low'].append(dff['last'])  I get an error message "ValueError: cannot reindex from a duplicate axis". The column 'new' should have these values: [0,5,10], [2,4,6], [5,10,15], [4,8,12] any help is very much appreciated!

Answer: Another possible solution: dff['new'] = dff['low'] + pd.Series([[x[1]] for x in dff['up']])  Output:  f d low up z new 0 a x [0, 5] [5, 10] [1, 3] [0, 5, 10] 1 a y [2, 4] [4, 6] [6, 2] [2, 4, 6] 2 b x [5, 10] [10, 15] [3, 7] [5, 10, 15] 3 b y [4, 8] [8, 12] [5, 10] [4, 8, 12]