eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to use OpenCV for a python project and for that reason want to install dlib library. I ran the command pip install dliband it gave me following error:  Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: dlib Building wheel for dlib (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for dlib (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] running bdist_wheel running build running build_ext ERROR: CMake must be installed to build dlib [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for dlib Failed to build dlib ERROR: Could not build wheels for dlib, which is required to install pyproject.toml-based projects  Since the error was Cmake must be installed I installed Cmake using pip install cmake but the errror still persisted. So I also installed cmake as mentioned here but the error still persist.

Answer: Since you are using anacoda you probably have messed up your run environment. Your error doesn't provide insight on what actually has happened but a makeshift, however it is demotivated, can fix your error. After installing cmake as mentioned in link answered install packages in root directory using sudo pip install cmake and sudo pip install dlib. Then you can import the modules in root directory using sys module as follows: import sys sys.path.insert(0, '/usr/path/to/modules/dlib')
Question:  data = {0: {'VAR1': 'A', 'VAR2': 'X', 'VAL1': 3, 'VAL2': 1}, 1: {'VAR1': 'A', 'VAR2': 'X', 'VAL1': 4, 'VAL2': 1}, 2: {'VAR1': 'A', 'VAR2': 'X', 'VAL1': 5, 'VAL2': 1}, 3: {'VAR1': 'A', 'VAR2': 'Y', 'VAL1': 3, 'VAL2': 2}, 4: {'VAR1': 'A', 'VAR2': 'Y', 'VAL1': 4, 'VAL2': 2}, 5: {'VAR1': 'A', 'VAR2': 'Y', 'VAL1': 5, 'VAL2': 2}, 6: {'VAR1': 'A', 'VAR2': 'Z', 'VAL1': 3, 'VAL2': 3}, 7: {'VAR1': 'A', 'VAR2': 'Z', 'VAL1': 4, 'VAL2': 3}, 8: {'VAR1': 'A', 'VAR2': 'Z', 'VAL1': 5, 'VAL2': 3}, 9: {'VAR1': 'B', 'VAR2': 'X', 'VAL1': 3, 'VAL2': 1}, 10: {'VAR1': 'B', 'VAR2': 'X', 'VAL1': 4, 'VAL2': 1}, 11: {'VAR1': 'B', 'VAR2': 'X', 'VAL1': 5, 'VAL2': 1}, 12: {'VAR1': 'B', 'VAR2': 'Y', 'VAL1': 3, 'VAL2': 2}, 13: {'VAR1': 'B', 'VAR2': 'Y', 'VAL1': 4, 'VAL2': 2}, 14: {'VAR1': 'B', 'VAR2': 'Y', 'VAL1': 5, 'VAL2': 2}, 15: {'VAR1': 'B', 'VAR2': 'Z', 'VAL1': 3, 'VAL2': 3}, 16: {'VAR1': 'B', 'VAR2': 'Z', 'VAL1': 4, 'VAL2': 3}, 17: {'VAR1': 'B', 'VAR2': 'Z', 'VAL1': 5, 'VAL2': 3}, 18: {'VAR1': 'C', 'VAR2': 'X', 'VAL1': 3, 'VAL2': 1}, 19: {'VAR1': 'C', 'VAR2': 'X', 'VAL1': 4, 'VAL2': 1}, 20: {'VAR1': 'C', 'VAR2': 'X', 'VAL1': 5, 'VAL2': 1}, 21: {'VAR1': 'C', 'VAR2': 'Y', 'VAL1': 3, 'VAL2': 2}, 22: {'VAR1': 'C', 'VAR2': 'Y', 'VAL1': 4, 'VAL2': 2}, 23: {'VAR1': 'C', 'VAR2': 'Y', 'VAL1': 5, 'VAL2': 2}, 24: {'VAR1': 'C', 'VAR2': 'Z', 'VAL1': 3, 'VAL2': 3}, 25: {'VAR1': 'C', 'VAR2': 'Z', 'VAL1': 4, 'VAL2': 3}, 26: {'VAR1': 'C', 'VAR2': 'Z', 'VAL1': 5, 'VAL2': 3}} df = pd.DataFrame.from_dict(dictio, orient='index')  I like to achieve:  new axes for every unique element in VAR1 new scatter plot of VAL1(x-value) and VAL2(y-value) for elements in VAR2 for every axes from VAR1  Example for axes of VAR1=A  I could not figure out how to do it with the groupby. My approach is not very good/correct: group_var1 = df.groupby('VAR1') for name_var1, grouped_var1 in group_var1: i = 0 fig, axes = plt.subplots(nrows=3, ncols=1,figsize=(20, 8), tight_layout=True) group_var2 = grouped_var1.groupby('VAR2') for name_var2, grouped_var2 in group_var2: grouped_var2.plot(kind='scatter', ax=axes[i], x='VAL1', y='VAL2') i+=1  EDIT: This works, but i highly dislike this approach group_var1 = df.groupby('VAR1') fig, axes = plt.subplots(nrows=3, ncols=1,figsize=(20, 8), tight_layout=True) i = 0 for name_var1, grouped_var1 in group_var1: group_var2 = grouped_var1.groupby('VAR2') for name_var2, grouped_var2 in group_var2: grouped_var2.plot(kind='scatter', ax=axes[i], x='VAL2', y='VAL1', c=['red','green','yellow']) i+=1 

Answer: I would use seaborn.relplot, which would work as a one-liner: import seaborn as sns sns.relplot(df, col='VAR1', hue='VAR2', x='VAL1', y='VAL2')  Output:
Question: I am currently working on a web scraping project and encountered an issue while scraping data from https://foundersfund.com/portfolio. I managed to retrieve all the links to each company's page successfully. However, upon testing some of these links, I noticed that the output HTML differs from what is shown in the inspect element tool. Consequently, I am unable to retrieve any information. import requests from bs4 import BeautifulSoup headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" } response = requests.get("https://foundersfund.com/company/figma/", headers=headers) soup = BeautifulSoup(response.content, "lxml") soup  The output is returning this: Google Colab I expected to retrieve information about Figma, but instead, I obtained information about SpaceX. Interestingly, when attempting to view pages for other companies, such as https://foundersfund.com/company/spotify/ or https://foundersfund.com/company/airbnb/, I encountered the same issue with SpaceX appearing instead. I have been troubleshooting this for several days and suspect that there may be an issue with the page itself. It seems that when I load the company's page, it briefly displays the SpaceX page before showing the requested company's page. Could someone please explain what might be happening here?

Answer: Content is loaded / rendered dynamically by JavaScript - Sometimes you can also see it on the page by refreshing it, then spacex is displayed for a short time, because the resource is first loaded and rendered. Only fractions of a second later the actual company information is rendered. So try to use the api instead for a single company: https://foundersfund.com/wp-json/wp/v2/company?slug=figma  or for the top 100: https://foundersfund.com/wp-json/wp/v2/company?per_page=100  Check your browsers dev tools for network traffic, to get an idea of how to find such information. Example import requests headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" } response = requests.get('https://foundersfund.com/wp-json/wp/v2/company?slug=figma', headers=headers).json() response
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'a': [10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 10, 22], 'b': [1, 1, 1, -1, -1, -1, -1, 2, 2, 2, 2, -1, -1, -1, -1], 'c': [25, 25, 25, 45, 45, 45, 45, 65, 65, 65, 65, 40, 40, 30, 30] } )  The expected output: Grouping df by c and a condition:  a b c 0 10 1 25 1 15 1 25 2 20 1 25 3 25 -1 45 4 30 -1 45 5 35 -1 45 6 40 -1 45 11 65 -1 40 12 70 -1 40  The process is as follows: a) Selecting the group that all of the b values is 1. In my data and this df there is only one group with this condition. b) Selecting first two groups (from top of df) that all of their b values are -1. For example: a) Group 25 is selected. b) There are three groups with this condition. First two groups are: Group 45 and 40. Note that there is a possibility in my data that there are no groups that has a or b condition. If that is the case, returning whatever matches the criteria is fine. For example the output could be only one group or no groups at all. The groups that I want are shown below:  These are my attempts that got very close: df1 = df.groupby('c').filter(lambda g: g.b.eq(1).all()) gb = df.groupby('c') new_gb = pd.concat([gb.get_group(group) for i, group in enumerate(gb.groups) if i < 2]) 

Answer: You can use custom masks for boolean indexing: # identify groups with all 1 m1 = df['b'].eq(1).groupby(df['c']).transform('all') # identify groups with all -1 m2 = df['b'].eq(-1).groupby(df['c']).transform('all') # keep rows of first 2 groups with all -1 m3 = df['c'].isin(df.loc[m2, 'c'].unique()[:2]) # select m1 OR m3 out = df[m1 | m3]  Or, for a variant without groupby, using set operations: # identify rows with 1/-1 m1 = df['b'].eq(1) m2 = df['b'].eq(-1) # drop c that have values other that 1/-1: {65} # drop -1 groups after 2nd occurrence: {30} drop = set(df.loc[~(m1|m2), 'c']) | set(df.loc[m2, 'c'].unique()[2:]) out = df[~df['c'].isin(drop)]  Output:  a b c 0 10 1 25 1 15 1 25 2 20 1 25 3 25 -1 45 4 30 -1 45 5 35 -1 45 6 40 -1 45 11 65 -1 40 12 70 -1 40  Intermediates (first approach):  a b c m1 m2 m3 0 10 1 25 True False False 1 15 1 25 True False False 2 20 1 25 True False False 3 25 -1 45 False True True 4 30 -1 45 False True True 5 35 -1 45 False True True 6 40 -1 45 False True True 7 45 2 65 False False False 8 50 2 65 False False False 9 55 2 65 False False False 10 60 2 65 False False False 11 65 -1 40 False True True 12 70 -1 40 False True True 13 10 -1 30 False True False 14 22 -1 30 False True False  Intermediates (second approach):  a b c m1 m2 ~isin(drop) 0 10 1 25 True False True 1 15 1 25 True False True 2 20 1 25 True False True 3 25 -1 45 False True True 4 30 -1 45 False True True 5 35 -1 45 False True True 6 40 -1 45 False True True 7 45 2 65 False False False 8 50 2 65 False False False 9 55 2 65 False False False 10 60 2 65 False False False 11 65 -1 40 False True True 12 70 -1 40 False True True 13 10 -1 30 False True False 14 22 -1 30 False True False
Question: I'd like to wrap all my tests with a fixture where the logs logged with loguru are checked for error messages. I tried this: @pytest.fixture(autouse=True) def assert_no_log_error(caplog): yield assert "ERROR" not in caplog.text  But caplog.text is always empty. I assume, caplog is cleared after the test and before the fixture actually checks the logs. How can I make this work? EDIT 1: I found an example about how to use loguru logs with pytest fixtures in the documentation of loguru: https://loguru.readthedocs.io/en/stable/resources/migration.html#replacing-caplog-fixture-from-pytest-library. However, this doesn't work either. It's really about that I want to check the logs in the fixture after the actual test, but I guess the captured logs are cleared after the test and before the end of the fixture.

Answer: I found the answer with the help of this post. First, since pytest uses the standard python logging module under the hood, the logs from loguru need to be captured properly. This can be done using the pytest-loguru module, according to the loguru docs. Install pytest-loguru with: pip install pytest-loguru Then the fixture can be written as: import logging from _pytest.logging import LogCaptureFixture @pytest.fixture(autouse=True) def assert_no_log_error(caplog: LogCaptureFixture): yield for record in caplog.get_records('call'): assert record.levelno < logging.ERROR
Question: New to Python. I come from a SQL world where I'm used to running queries and applying them. It's handy to take a list of things, get their count and then use a subset of that count (like the top 5) and apply it to other data. With Python/Pandas, I still have not quite grokked the process. By way of example: A simple dataset: `import pandas as pd dataset = ( [1,2,3,4,5,6], [1,None,3,4,5,6], [1,None,3,4,5,6], [1,2,None,4,5,6], [1,None,3,None,5,6], [1,2,None,4,5,6], [1,None,3,None,5,6], [1,2,3,4,5,None], [1,2,3,4,5,None] ) df = pd.DataFrame(dataset, columns=['A','B','C','D','E','F'])`  Then make a dataframe to find the NaNs:  nan_df = df.isna() Then count the instances of each row:  grouped_nan = nan_df.groupby(['A','B', 'C', 'D','E', 'F'], sort=True).value_counts() The original set I was working was had ~200 rows. This simplified example yields this: A B C D E F False False False False False False 1 True 2 True False False False 2 True False False False False 2 True False False 2 Name: count, dtype: int64  This is where I run into trouble. The things I want to do are best done in a Dataframe (the above is a Series). The following makes it a DataFrame:  grouped_nan_df = grouped_nan.to_frame() But it doesn't bring along the last column (the count) in a way I can manage. I can see it, but I can't do anything with it. If I try to reference the column with the counts, it does not recognize it. If I try to rename that last column it doesn't work: `grouped_nan_df.rename(columns={grouped_nan_df.columns[5]:"new_count"}, inplace=True)`  gives the error "index 5 is out of bounds for axis 0 with size 1". What I want at the end is a DataFrame that includes the counts. Is there a way to get there? Any help appreciated! Andy

Answer: IIUC, you can just use groupby with as_index=False and then take the group size: out = nan_df.groupby(['A','B', 'C', 'D','E', 'F'], as_index=False).size()  Output:  A B C D E F size 0 False False False False False False 1 1 False False False False False True 2 2 False False True False False False 2 3 False True False False False False 2 4 False True False True False False 2
Question: I have 2 dataframes, one that contains data and one that contains exlusions that need to be merged onto the data and marked as included(True or False). I have been doing this as follows for a couple of years by simply adding a new column to the exclusions data frame and setting everything to True, then merging that onto the main dataframe which results in the additional column containing either True or NaN. Finally I run a pd.fillna to replace all the NaN values with False and I'm good to go. import pandas as pd MainData = {'name': ['apple', 'pear', 'orange', 'watermelon'], 'other': ['blah' , 'blah', 'blah' , 'blah']} dfMainData = pd.DataFrame(MainData) Exclusions = {'name': ['pear' , 'watermelon'], 'reason': ['pears suck', 'too messy!']} dfExclusions = pd.DataFrame(Exclusions) dfExclusions['excluded'] = True dfMainData = pd.merge(dfMainData, dfExclusions, how='left', on='name') dfMainData['excluded'] = dfMainData['excluded'].fillna(False)  I was previously running pands 1.2.4 but am making code updates and am migrating to 2.2.1 as part of this, and am now receiving the following warning:  dfMainData['excluded'] = dfMainData['excluded'].fillna(False) :1: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set pd.set_option('future.no_silent_downcasting', True)  It still technically works but it appears that this is NOT the pandas'esque way of doing things, so I am curious how I should be going about this now to avoid compatibility issues in the future?

Answer: You can use (for pandas 2.2.1 etc) : dfMainData['excluded'] = dfMainData['excluded'].fillna(0).astype('bool')  which gives  name other reason excluded 0 apple blah NaN False 1 pear blah pears suck True 2 orange blah NaN False 3 watermelon blah too messy! True
Question: I need to creat an empty data frame that stores values from a column of another data frame base on some conditions being met in two columns of the same second data frame. I have a data frame test_mob_df = pd.DataFrame( {"geoid_o": [10002, 18039, 18039, 18182, 10006, 18111, 18005, 17001], "geoid_d": [10005, 18039, 18111, 18182, 18005, 17004, 18050, 15001], "pop_flows": [20,10,9,15,2,1,6,30]}) and a list of ids that I am interested in as state_county_fip = [18182, 18111, 18005, 18039, 18050, 18001]. I now need to create a new $nxn$ data frame (or an array) with row and column names as sorted state_county_fips, that stores the values in the pop_flows column of $test_mob_df$ whenever values in the same row of columns geoid_o and geoid_d match or not. In essence, the resulting data frame should look like this:     18005 18039 18005 18050 18111 18182     18005 0 0 0 0 0 0   18039 0 10 0 0 9 0   18005 0 0 0 6 0 0   18050 0 0 0 0 0 0   18111 0 0 0 0 0 0   18182 0 0 0 0 0 15    That is, I need to create a a dataframe (or matrix) of population flows from geoid_o to geoid_d, and when we do not have population flow from geoid_o to geoid_d, then we asign zero to the corresponding cell. For instance, 10 individuals moved from geoid_o 18005 to geoid_d 18050. I can't seem to figure out how to do this beyond using query to create a data frame (from test_mob_df) that has the geoids of interest: data_counties_of_interest = test_mob_df.query("18001<=geoid_o<18200 and 18001<=geoid_d<18200"). I will deeply appreciate any help you can offer.

Answer: You can do: test_mob_df = pd.DataFrame( { "geoid_o": [10002, 18039, 18039, 18182, 10006, 18111, 18005, 17001], "geoid_d": [10005, 18039, 18111, 18182, 18005, 17004, 18050, 15001], "pop_flows": [20, 10, 9, 15, 2, 1, 6, 30], } ) state_county_fip = [18182, 18111, 18005, 18039, 18050, 18001] out = pd.crosstab( test_mob_df.loc[test_mob_df["geoid_o"].isin(state_county_fip), "geoid_o"], test_mob_df.loc[test_mob_df["geoid_d"].isin(state_county_fip), "geoid_d"], values=test_mob_df["pop_flows"], aggfunc="first", ) out = ( out.reindex(index=state_county_fip, columns=state_county_fip) .fillna(0) .sort_index(axis=1) .sort_index() .astype(int) ) print(out)  Prints: geoid_d 18001 18005 18039 18050 18111 18182 geoid_o 18001 0 0 0 0 0 0 18005 0 0 0 6 0 0 18039 0 0 10 0 9 0 18050 0 0 0 0 0 0 18111 0 0 0 0 0 0 18182 0 0 0 0 0 15
Question: Given this Polars DataFrame: df = pl.DataFrame( { "id": [1, 2, 3, 4, 5], "values": ["A", "B", "A--B", "C--A", "D"], } )  1, How can I group/cluster it so that 1,2 and 3 ends up in the same group? 2. Can I even achieve having 4 in the same group/cluster?

Answer: Assuming you want to merge groups based on the substrings (separated by --), this is unfortunately not straightforward. You can't vectorize this since a member of a group can link to another group that links to another, etc. One option is to use graph theory to identify the connected components. You can do this with networkx and connected_components: import networkx as nx G = nx.from_pandas_edgelist(df.with_columns(pl.col('values').str.split('--')) .explode('values'), source='id', target='values') S = set(df['id']) mapper = {n: i for i, c in enumerate(nx.connected_components(G)) for n in c&S} # {1: 0, 2: 0, 3: 0, 4: 0, 5: 1} out = df.group_by(pl.col('id').replace(mapper).alias('group')).agg(pl.all())  Output: shape: (2, 3) ┌───────┬─────────────┬──────────────────────┐ │ group ┆ id ┆ values │ │ --- ┆ --- ┆ --- │ │ i64 ┆ list[i64] ┆ list[str] │ ╞═══════╪═════════════╪══════════════════════╡ │ 0 ┆ [1, 2, … 4] ┆ ["A", "B", … "C--A"] │ │ 1 ┆ [5] ┆ ["D"] │ └───────┴─────────────┴──────────────────────┘  Graph:
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'a': [10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70], 'b': [1, 1, 1, -1, -1, -2, -1, 2, 2, -2, -2, 1, -2], } )  The mask is: mask = ( (df.b == -2) & (df.b.shift(1) > 0) )  Expected output: slicing df up to the first instance of the mask:  a b 0 10 1 1 15 1 2 20 1 3 25 -1 4 30 -1 5 35 -2 6 40 -1 7 45 2 8 50 2  The first instance of the mask is at row 9. So I want to slice the df up to this index. This is what I have tried. It works but I am not sure if it is the best way: idx = df.loc[mask.cumsum().eq(1) & mask].index[0] result = df.iloc[:idx] 

Answer: You can filter by inverted mask with Series.cummax: out = df[~mask.cummax()] print (out) a b 0 10 1 1 15 1 2 20 1 3 25 -1 4 30 -1 5 35 -2 6 40 -1 7 45 2 8 50 2  How it working: print (df.assign(mask=mask, cumax=mask.cummax(), inv_cummax=~mask.cummax())) a b mask cumax inv_cummax 0 10 1 False False True 1 15 1 False False True 2 20 1 False False True 3 25 -1 False False True 4 30 -1 False False True 5 35 -2 False False True 6 40 -1 False False True 7 45 2 False False True 8 50 2 False False True 9 55 -2 True True False 10 60 -2 False True False 11 65 1 False True False 12 70 -2 True True False
Question: I have two data frames, which should be concatenated. Both are multi index data frames with identical indexes, but in a different order. So, the index of the first data frame (df) looks like: MultiIndex([(11, 1, 1), (11, 1, 2), (11, 1, 3), ... (11, 24, 5), (11, 24, 6), (11, 24, 7)], names=['id_a', 'id_b', 'id_c'], length=168)  The second looks like: MultiIndex([(11, 1, 1), (11, 2, 1), (11, 3, 1), (11, 3, 2), ... (11, 5, 23), (11, 6, 23), (11, 7, 23), (11, 7, 24)], names=['id_a', 'id_c', 'id_b'], length=168)  As you can see, indexes are in a different order. Now by running pd.concat([df, df2]).index.names I get the following results: FrozenList(['id_a', None, None])  How to reproduce import pandas as pd # create first data frame idx = pd.MultiIndex.from_product( [['A1', 'A2', 'A3'], ['B1', 'B2', 'B3'], ['C1', 'C2', 'C3']], names=['a', 'b', 'c']) cols = ['2010', '2020'] df = pd.DataFrame(1, idx, cols) # Create second data frame with varying order idx = pd.MultiIndex.from_product( [['A1', 'A2', 'A3'], ['C1', 'C2', 'C3'], ['B1', 'B2', 'B3']], names=['a', 'c', 'b']) df2 = pd.DataFrame(2, idx, cols) result = pd.concat([df, df2])  Output > df 2010 2020 a b c A1 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 A2 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 A3 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 > df2 2010 2020 a c b A1 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 A2 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 A3 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 > result 2010 2020 a A1 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 A2 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 A3 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 B3 C1 1 1 C2 1 1 C3 1 1 A1 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 A2 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 A3 C1 B1 2 2 B2 2 2 B3 2 2 C2 B1 2 2 B2 2 2 B3 2 2 C3 B1 2 2 B2 2 2 B3 2 2 > result.index.names FrozenList(['a', None, None])  Indexes 'b' and 'c' are gone.

Answer: If your levels are in a different order, you first need to reorder them with reorder_levels: result = pd.concat([df, df2.reorder_levels(df.index.names)])  If you have more than 2 DataFrames to concatenate: dfs = [df, df2, df3, df4] levels = dfs[0].index.names result = pd.concat([d.reorder_levels(levels) for d in dfs])  Output:  2010 2020 a b c A1 B1 C1 1 1 C2 1 1 C3 1 1 B2 C1 1 1 C2 1 1 C3 1 1 ... A1 B1 C1 2 2 B2 C1 2 2 B3 C1 2 2 B1 C2 2 2 ...
Question: I have a large datasets need to filter out root packages as below:  sort data(package column) by string length. start from beginning, scan the following data, if it starts with current data, then mark it as False. repeat step 2 till end.  To improve performance, I add a flag column to keep track it's processed or not. Sound like now the big-O is n instead of n square. anything we can improve? The obvious one is the second for loop, now use continue to skip previous items, maybe more better way do do that? import re import pandas as pd import tabulate def dumpdf(df): if len(df) == 0: return df = df.reset_index(drop=True) tab = tabulate.tabulate(df, headers='keys', tablefmt='psql',showindex=True) print(tab) return def main(): data = [ ['A','com.example'], ['A','com.example.a'], ['A','com.example.b.c'], ['A','com.fun'], ['B','com.demo'], ['B','com.demo.b.c'], ['B','com.fun'], ['B','com.fun.e'], ['B','com.fun.f.g'] ] df = pd.DataFrame(data,columns=['name','package']) df ['flag'] = None df = df.sort_values(by="package", key=lambda x: x.str.len()).reset_index(drop=True) for idx,row in df.iterrows(): if row['flag'] == None: df.loc[idx,'flag'] = True for jdx, jrow in df.iterrows(): if jdx <= idx: continue if row['name'] == jrow['name']: if jrow['package'].startswith(row['package']): df.loc[jdx,'flag'] = False # df = df[df['flag']] df = df.groupby('name',as_index=False).agg({'package':'\n'.join}) dumpdf(df) return main()  

Answer: IIUC you can do: from functools import cmp_to_key from itertools import groupby def fn(g): out = [] for _, k in groupby(g.sort_values(), cmp_to_key(lambda a, b: not b.startswith(a))): out.append(next(k)) return pd.Series(out) out = df.groupby("name")["package"].apply(fn).droplevel(1).reset_index() print(out)  Prints:  name package 0 A com.example 1 A com.fun 2 B com.demo 3 B com.fun