eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I know this question has been answered but my use case is slightly different. I am trying to setup a regex pattern to split a few strings into a list. Input Strings: 1. "ABC-QWERT01" 2. "ABC-QWERT01DV" 3. "ABCQWER01"  Criteria of the string ABC - QWERT 01 DV 1 2 3 4 5  The string will always start with three chars The dash is optional there will then be 3-10 chars Left padded 0-99 digits the suffix is 2 chars and is optional  Expected Output 1. ['ABC','-','QWERT','01'] 1. ['ABC','-','QWERT','01', 'DV'] 1. ['ABC','QWER','01','DV']  I have tried the following patterns a bunch of different ways but I am missing something. My thought was start at the beginning of the string, split after the first three chars or the dash, then split on the occurrence of two decimals. Pattern 1: r"([ -?, \d{2}])+" This works but doesn't break up the string by the first three chars if the dash is missing Pattern 2: r"([^[a-z]{3}, -?, \d{2}])+" This fails as a non-pattern match, nothing gets split Pattern 3: r"([^[a-z]{3}|-?, \d{2}])+" This fails as a non-pattern match, nothing gets split Any tips or suggestions?

Answer: You can use a pattern similar to : (?i)([A-Z]{3})(-?)([A-Z]*)([0-9]{2})([A-Z]*)  Code: import re def _parts(s): p = r'(?i)([A-Z]{3})(-?)([A-Z]*)([0-9]{2})([A-Z]*)' return re.findall(p, s) print(_parts('ABC-QWERT01DV')) print(_parts('ABCQWER01')) print(_parts('ABC-QWERT01'))  Prints [('ABC', '-', 'QWERT', '01', 'DV')] [('ABC', '', 'QWER', '01', '')] [('ABC', '-', 'QWERT', '01', '')]   Notes:  (?i): insensitive flag. ([A-Z]{3}): capture group 1 with any 3 letters. (-?): capture group 2 with an optional dash. ([A-Z]*): capture group 3 with 0 or more letters. ([0-9]{2}): capture group 4 with 2 digits. ([A-Z]*): capture group 5 with 0 or more letters.
Question: I'm trying to install the quikcfix library on my windows machine. The python version is 3.12.2. However I get the below error. python setup.py bdist_wheel did not run successfully exit code: 1 [7 lines of output] Testing for std::tr1::shared_ptr... ...not found Testing for std::shared_ptr... ...not found Testing for std::unique_ptr... ...not found error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools  I installed the Microsoft C++ build tools as well. But it still gives the same error. Can anyone suggest a solution for this error

Answer: Download the quickfix from here: https://github.com/kazcfz/QuickFIX-prebuilt-wheel What I see from the link, the latest version of python which supports quickfix is 3.9 It will not work for python 3.12
Question: I am working with the meta-feature extractor package: pymfe for complexity analysis. On a small dataset, this is not a problem, for example. pip install -U pymfe from sklearn.datasets import make_classification from sklearn.datasets import load_iris from pymfe.mfe import MFE data = load_iris() X= data.data y = data.target extractor = MFE(features=[ "t1"], groups=["complexity"], summary=["min", "max", "mean", "sd"]) extractor.fit(X,y) extractor.extract() (['t1'], [0.12])  My dataset is large (32690, 80) and this computation gets killed for exessive memory usage. I work on Ubuntu 24.04 having 32GB RAM. To reproduce scenario: # Generate the dataset X, y = make_classification(n_samples=20_000,n_features=80, n_informative=60, n_classes=5, random_state=42) extractor = MFE(features=[ "t1"], groups=["complexity"], summary=["min", "max", "mean", "sd"]) extractor.fit(X,y) extractor.extract() Killed  Question: How do I split this task to compute on small partitions of the dataset, and combine final results (averaging)?

Answer: Managed to find a workaround. # helper functions def split_dataset(X, y, n_splits): # data splits split_X = np.array_split(X, n_splits) split_y = np.array_split(y, n_splits) return split_X, split_y def compute_meta_features(X, y): # meta-features for a partition extractor = MFE(features=["t1"], groups=["complexity"], summary=["min", "max", "mean", "sd"]) extractor.fit(X, y) return extractor.extract() def average_results(results): # summary of results features = results[0][0] summary_values = np.mean([result[1] for result in results], axis=0) return features, summary_values # Split dataset n_splits = 10 # ten splits split_X, split_y = split_dataset(X, y, n_splits) # meta-features results = [compute_meta_features(X_part, y_part) for X_part, y_part in zip(split_X, split_y)] # Combined results final_features, final_summary = average_results(results)
Question: I need to make a map of distances to the nearest object. I have a solution where i am looping over every point of a map, and every object, calculating the distance to all of them, and then leaving only minimum distance. The problem here is that if I am woking with real data, the map can easily contain 10s of millions of points, and there can be more than 100 objects. Is there any better code implementation for solving this problem? Loading packages import pandas as pd import numpy as np import matplotlib.pyplot as plt  Generate synthetic map coord_dict = {"X": [], "Y": []} for x_value in range(0, 10000, 50): for y_value in range(0, 5000, 50): coord_dict["X"].append(x_value) coord_dict["Y"].append(y_value) map_df = pd.DataFrame(coord_dict)  Generate points to calculate distance from well_points_dict = {"X": [500, 1500, 4000, 5500, 6250, 7500, 8000, 9000], "Y": [500, 4000, 2000, 1500, 500, 5000, 100, 2500]} wells_df = pd.DataFrame(well_points_dict)  Calculate distances calculations_count = 0 distance_map = np.zeros(map_df.shape) for i in range(map_df.shape[0]): d = [] for j in range(wells_df.shape[0]): d.append(((map_df["X"].iloc[i]-wells_df["X"][j])**2 + (map_df["Y"].iloc[i]- wells_df["Y"][j])**2)**0.5) calculations_count += 1 dd = min(d) distance_map[i,1] = dd # print(calculations_count)  Print resulting map plt.figure(figsize=(10,10)) plt.scatter(x=map_df["X"],y=map_df["Y"],c=distance_map[:,1],s=1,cmap='terrain') for i in range(len(wells_df)): plt.plot(wells_df["X"][i],wells_df["Y"][i], color='black', marker='o',markersize=3) plt.title('Calculated map') plt.xlabel('X') plt.ylabel('Y') plt.axis('scaled') plt.tight_layout() plt.colorbar(shrink=0.25)  Result map example: 

Answer: KDTree is what you are looking for, there is an implementation of it in scipy.spatial. import numpy as np import matplotlib.pyplot as plt from scipy import spatial  Given your trial points: x_value = np.arange(0, 10000, 50) y_value = np.arange(0, 5000, 50) X, Y = np.meshgrid(x_value, y_value) points = np.stack([X.ravel(), Y.ravel()]).T  And well points: x_well = np.array([500, 1500, 4000, 5500, 6250, 7500, 8000, 9000]) y_well = np.array([500, 4000, 2000, 1500, 500, 5000, 100, 2500]) wells = np.stack([x_well, y_well]).T  We can create a KDTree: interpolator = spatial.KDTree(wells)  And query efficiently the tree to get distances and also indices of which point it is closer: distances, indices = interpolator.query(points) # 7.12 ms ± 711 µs per loop (mean ± std. dev. of 30 runs, 100 loops each)  Plotting the result leads to: fig, axe = plt.subplots() axe.scatter(*points.T, marker=".", c=distances) axe.scatter(*wells.T, color="black") axe.grid()   We see the Voronoi diagram appearing on the color map which is a good confirmation distances are correctly interpreted wrt reference points (wells): voronoi = spatial.Voronoi(wells) # ... spatial.voronoi_plot_2d(voronoi, ax=axe)   Where the object Voronoi is the Voronoi diagram based on your reference points (wells) and voronoi_plot_2d an helper to draw it on axes.
Question: While creating my website, I face the following problem: a user can add an article, and so the article model has a ForeignKey field - author. I created a form for adding it, inheriting from ModelForm, since I think it is more logical that way (as it interacts directly with db). The problem is that I cannot add this field to the Form, as it cannot be filled; it has to be taken from the request itself. author = request.user.pk  So how can I make a form save the author id as well? Without it, there's an error: NOT NULL CONSTRAINT FAILED, which is logical as the form doesn't save pk and so it is null; but a FOREIGN KEY cannot be null. The only way is to inherit it from class Form? I don't really want to do it... I was thinking about 'overriding' method save() so that it has one more argument - author id: form.save(author=request.user.pk)  This way it would work. But I changed my mind, because if something goes wrong, this will mess up the whole database... The save() method is too global. What's more, there might well be another way to solve my problem, which is more effective and clear. Here's the code for my form: class ArticleForm(ModelForm): class Meta: model = Article fields = ['title', 'content'] widgets = {'title' : TextInput(attrs={'class' : 'create_article_title', 'placeholder' : 'Enter the heading...' }), 'content' : Textarea(attrs={'class' : 'create_article_content', 'placeholder' : 'Your article...' })}  and my model: class Article(Model): title = CharField(max_length=30) content = CharField(max_length=5000) author = ForeignKey(User, on_delete=CASCADE) rating = IntegerField(default=0) created = DateField(auto_now_add=True) last_updated = DateField(auto_now_add=True) articles = ArticleManager() objects = Manager()  and my view: class AddArticle(View): template_name = 'myapp/addarticle.html' def get(self, request): context = {'add_article_form' : ArticleForm()} return render(request, self.template_name, context=context) def post(self, request): form = ArticleForm() form.save() return redirect('index')  It seems that I completely misunderstand the basics of ModelForm creation...

Answer: There are other ways. Such as adding a hidden field to your form and set its value as the request.user.id. Although, I must say that your initial thought related to the save() method is the correct one. However, not overriding it but using an optional keyword provided by the method itself. Also, do not forget to set form data and validate it, which is one of the main features: class AddArticle(View): ... def post(self, request): form = ArticleForm(request.POST) if form.is_valid(): instance = form.save(commit=False) instance.author = request.user instance.save() return redirect('index')
Question: I'm trying to assign multiple entries from a list per date to my dateframe, so for each day, I would have all the values from the list. I tried passing the list as an argument; however, I'm having problems with the length of the dataframe. Conceptual code: from datetime import datetime import polars as pl Value_list = ["xd","xd1"] dates_df= pl.DataFrame( { "dates": pl.datetime_range( start=datetime(2023, 1, 1), end=datetime(2023, 12, 10), interval="1d", eager=True, closed="both", ), "Producto": Value_list } ) dates_df  My actual code: from datetime import datetime import polars as pl dates_df= pl.DataFrame( { "dates": pl.datetime_range( start=datetime(2023, 1, 1), end=datetime(2023, 12, 10), interval="1d", eager=True, closed="both", ), "Producto": "xd" } ) dates_df  Expected Output: shape: (344, 2) ┌─────────────────────┬──────────┐ │ dates ┆ Producto │ │ --- ┆ --- │ │ datetime[μs] ┆ str │ ╞═════════════════════╪══════════╡ │ 2023-01-01 00:00:00 ┆ xd │ │ 2023-01-01 00:00:00 ┆ xd1 │ │ 2023-01-02 00:00:00 ┆ xd │ │ 2023-01-02 00:00:00 ┆ xd1 │ │ 2023-01-03 00:00:00 ┆ xd │ │ … ┆ … │ │ 2023-12-08 00:00:00 ┆ xd1 │ │ 2023-12-09 00:00:00 ┆ xd │ │ 2023-12-09 00:00:00 ┆ xd1 │ │ 2023-12-10 00:00:00 ┆ xd │ │ 2023-12-10 00:00:00 ┆ xd1 │ └─────────────────────┴──────────┘ 

Answer: Are you looking for a cross-join? from datetime import datetime import polars as pl dates = pl.datetime_range( start=datetime(2023, 1, 1), end=datetime(2023, 12, 10), interval="1d", eager=True, closed="both", ).to_frame('date') values = pl.Series('producto', ['xd', 'xd1']).to_frame() print(dates.join(values, how='cross'))  output: shape: (688, 2) ┌─────────────────────┬──────────┐ │ date ┆ producto │ │ --- ┆ --- │ │ datetime[μs] ┆ str │ ╞═════════════════════╪══════════╡ │ 2023-01-01 00:00:00 ┆ xd │ │ 2023-01-01 00:00:00 ┆ xd1 │ │ 2023-01-02 00:00:00 ┆ xd │ │ 2023-01-02 00:00:00 ┆ xd1 │ │ 2023-01-03 00:00:00 ┆ xd │ │ … ┆ … │ │ 2023-12-08 00:00:00 ┆ xd1 │ │ 2023-12-09 00:00:00 ┆ xd │ │ 2023-12-09 00:00:00 ┆ xd1 │ │ 2023-12-10 00:00:00 ┆ xd │ │ 2023-12-10 00:00:00 ┆ xd1 │ └─────────────────────┴──────────┘
Question: Help me in finding out the exact difference between time and timeit magic commands in jupyter notebook. %time a="stack overflow" time.sleep(2) print(a)   It shows only some micro seconds, But I added sleep time as 2 seconds 

Answer: Magics that are prefixed with double percentage signs (%%) are considered cell magics. This differs from line magics which are prefixed with a single percentage sign (%). It follows that,  time is used to time the execution of code either at a line level (%) or cell level (%%). timeit is similar but runs the code multiple times to provide more accurate and statistically significant timing information.  The key distinction is that %time and %%time (docs) measure the execution time of code once, while %timeit and %%timeit (docs) measure the execution time of code multiple times to give an average time and standard deviation. Hence, %%time a="stack overflow" time.sleep(2) print(a)  This output shows both the CPU time and the wall time. The wall time includes the 2 seconds sleep time, while the CPU time accounts for the actual processing time. Replacing %%time with %time will show the CPU time and the wall time for the next single line that is executed.
Question: I am working on an application that needs to provide the count of certain entries in a dataframe. Am I missing something that is not rendering the required outcome? Input: | Release | Mapping | Coding | |-----------|---------|--------| | release_a | A1 | C2 | | release_c | A1 | C2 | | release_a | A1 | C2 | | release_a | A1 | C1 | | release_b | B | C1 | | release_c | B | C2 | | release_c | B | C3 | | release_a | C | C1 | | release_c | A1 | C1 | | release_c | A1 | C3 | | release_a | C | C1 |  Outcome expected: | Release | Mapping | |-----------|--------------| | release_a | A1 - 3, C-2 | | release_b | B-1 | | release_c | A1 -3, B - 2 |  Code used: df.groupby(['Release', 'Mapping'])['Coding'].agg(count='count')  What I am getting:  Maybe I don't have a thorough understanding of how use the agg method. If there is any better alternative, please suggest it.

Answer: Here's an approach with collections.Counter: from collections import Counter out = (df.groupby('Release')['Mapping'] .agg(lambda x: ", ".join(f"{k} - {v}" for k, v in Counter(x).items())) .reset_index() )  Output:  Release Mapping 0 release_a A1 - 3, C - 2 1 release_b B - 1 2 release_c A1 - 3, B - 2  Or via Series.value_counts, but that seems to be slower: out2 = (df.groupby('Release')['Mapping'] .agg(lambda x: ", ".join(f"{k} - {v}" for k, v in x.value_counts() .items())) .reset_index() ) # out.equals(out2) # True  Already on a small set like this, both will outperform using df.iterrows (and a double groupby) as suggested in the answer by @VinodBaste. Avoid it where you can, and in general you can (see here). Timings: # Counter 747 µs ± 25.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) # value_counts 1.27 ms ± 40.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) # iterrows 2.45 ms ± 48.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  Data used import pandas as pd data = { 'Release': ['release_a', 'release_c', 'release_a', 'release_a', 'release_b', 'release_c', 'release_c', 'release_a', 'release_c', 'release_c', 'release_a'], 'Mapping': ['A1', 'A1', 'A1', 'A1', 'B', 'B', 'B', 'C', 'A1', 'A1', 'C'], 'Coding': ['C2', 'C2', 'C2', 'C1', 'C1', 'C2', 'C3', 'C1', 'C1', 'C3', 'C1'] } df = pd.DataFrame(data)
Question: My program generates a data_list using a while loop (the code for generating the data_list is not important, and I have attached the main code below). At the end of the loop, after the data_list is generated, I want to write this data_list to the test.xlsx file. However, for each new data_list to be saved in a new tab (that is, if I had 10 iterations in the loop, there would be 10 different tabs in the test.xlsx file). At the moment, my code works in such a way that each new tab overwrites the previous one, that is, at the end of the loop, there is one tab in the test.xlsx file. Tell me how to do it in such a way that each new generated data_list is written in a new tab. start=0 finish=1 datas = response.json() while finish < 11: data = datas[start:finish] data_list = [] # some code for generating the data_list df = pd.DataFrame(data_list) writer = pd.ExcelWriter('test.xlsx') df.to_excel(writer, sheet_name=f'{finish}', index=False) writer._save() finish +=1 start +=1 

Answer: Instead of saving each sheet use with to create the ExcelWriter with pd.ExcelWriter('test.xlsx') as writer: while finish < 11: data = datas[start:finish] data_list = [] df = pd.DataFrame(data_list) df.to_excel(writer, sheet_name=f'{finish}', index=False) finish += 1 start += 1
Question: Need to extract sequence of negative values where earlier negative value is smaller than current value and next value is smaller than current value import pandas as pd # Create the DataFrame with the given values data = { 'Value': [0.3, 0.2, 0.1, -0.1, -0.2, -0.3, -0.4, -0.35, -0.25, 0.1, -0.15, -0.25, -0.13, -0.1, 1] } df = pd.DataFrame(data) print("Original DataFrame:") print(df)  My Code: # Initialize a list to hold the sequences sequences = [] current_sequence = [] # Iterate through the DataFrame to apply the condition for i in range(1, len(df) - 1): prev_value = df.loc[i - 1, 'Value'] curr_value = df.loc[i, 'Value'] next_value = df.loc[i + 1, 'Value'] # Check the condition if curr_value < prev_value and curr_value < next_value: current_sequence.append(curr_value) else: # If the current sequence is not empty and it's a valid sequence, add it to sequences list and reset if current_sequence: sequences.append(current_sequence) current_sequence = [] # Add the last sequence if it's not empty if current_sequence: sequences.append(current_sequence)  My Output: Extracted Sequences: [-0.4] [-0.25]  Expected Output: [-0.1,-0.2,-0.3,-0.4] [-0.15,-0.25] 

Answer: You can build masks to identify the negative values and consecutive decreasing values and use groupby to split: # is the value negative? m1 = df['Value'].lt(0) # is the value decreasing? m2 = df['Value'].diff().le(0) m = m1&m2 # aggregate out = df[m].groupby((~m).cumsum())['Value'].agg(list).tolist()  Output: [[-0.1, -0.2, -0.3, -0.4], [-0.15, -0.25]]  If you just want to filter: out = df[m]  Output:  Value 3 -0.10 4 -0.20 5 -0.30 6 -0.40 10 -0.15 11 -0.25  Intermediates:  Value m1 df['Value'].diff() m2 m1&m2 0 0.30 False NaN False False 1 0.20 False -0.10 True False 2 0.10 False -0.10 True False 3 -0.10 True -0.20 True True 4 -0.20 True -0.10 True True 5 -0.30 True -0.10 True True 6 -0.40 True -0.10 True True 7 -0.35 True 0.05 False False 8 -0.25 True 0.10 False False 9 0.10 False 0.35 False False 10 -0.15 True -0.25 True True 11 -0.25 True -0.10 True True 12 -0.13 True 0.12 False False 13 -0.10 True 0.03 False False 14 1.00 False 1.10 False False
Question: I'm implementing a discrete inverse Fourier transform in Python to approximate the inverse Fourier transform of a Gaussian function.  The input function is sqrt(pi) * e^(-w^2/4) so the output must be e^(-x^2). While the shape of the resulting function looks correct, the x-axis scaling seems to be off (There might be just some normalization issue). I expect to see a Gaussian function of the form e^(-x^2), but my result is much narrower. This is my implementation: import matplotlib.pyplot as plt import numpy as np from sympy import symbols, exp, pi, lambdify, sqrt # Defining the Fourier transform of a Gaussian function, sqrt(pi) * exp(-omega ** 2 / 4) x, omega = symbols('x omega') f_gaussian_symbolic = exp(-omega ** 2 / 4) * sqrt(pi) f_gaussian_function = lambdify(omega, f_gaussian_symbolic, 'numpy') def fourier_inverse(f, n, omega_max): """ This function computes the inverse Fourier transform of a function f. :param f: The function to be transformed :param n: Number of samples :param omega_max: The max frequency we want to be sampled """ omega_range = np.linspace(-omega_max, omega_max, n) f_values = f(omega_range) inverse_f = np.fft.ifftshift(np.fft.ifft(np.fft.fftshift(f_values))) delta_omega = omega_range[1] - omega_range[0] x_range = np.fft.ifftshift(np.fft.fftfreq(n, d=delta_omega)) inverse_f *= delta_omega * n / (2 * np.pi) return x_range, inverse_f plt.figure(figsize=(10, 5)) x_range, inverse_f = fourier_inverse(f_gaussian_function, 10000, 100) plt.plot(x_range, inverse_f.real) plt.ylim(-2, 2) plt.xlim(-4, 4) plt.show()  I expect the plot to be:  Buy my output is this:  The shape of the function looks correct, but it's much narrower than expected. I suspect there's an issue with how I'm calculating or scaling the x_range in my fourier_inverse function. What am I doing wrong in my implementation, and how can I correct the x-axis scaling to get the expected Gaussian function e^(-x^2)?

Answer: It looks like you're using frequency in the x-axis when you expect angular frequency. You should modify your x_range computation like this:  x_range = 2 * np.pi * np.fft.ifftshift(np.fft.fftfreq(n, d=delta_omega))  With this change, the resulting plot looks like this:
Question: data1 = { 'alias_cd': ['12345', '12345', '12345'], 'country_cd': ['AU', 'AU', 'AU2'], 'pos_name': ['st1', 'Jh', 'Jh'], 'ts_allocated': [100, 100, 100], 'tr_id': ['None', 'None', 'None'], 'ty_name': ['E2E', 'E2E', 'E2E'] } data2 = { 'alias_cd': ['12345', '12345'], 'country_cd': ['AU', 'AU3'], 'pos_name': ['st1', 'st2'], 'ts_allocated': [200, 100], 'tr_id': ['None', 'None'], 'ty_name': ['E2E', 'E2E'] } df1 = pd.DataFrame(data1) df2 = pd.DataFrame(data2)  output should be  alias_cd country_cd pos_name ts_allocated tr_id ty_name etl_flag 1 12345 AU st1 200 None E2E U 2 12345 AU3 st2 100 None E2E D 3 12345 AU st1 100 None E2E I 4 12345 AU Jh 100 None E2E I 5 12345 AU2 Jh 100 None E2E I  Because: The combination of alias_cd and country_cd acts as a primary key. 1.If a combination exists in df2 and df1 (12345 AU), it will be marked for 'Update' in df2, and all corresponding rows in df1 for the same combination will be marked as 'Insert'. for the above example for 12345 AU records in df2 will be etl_flag= 'Update' and add records for the same combination from df1 to df2 with etl_flag as 'Insert' 2.12345 AU3 exists in df2 but not in df1, so it will be tagged as 'DELETE' in the etl_flag column. 3.Any new combination that appears in df1 and not present in df2 will be tagged as 'Insert' in the etl_flag column. How can I achieve this efficiently? This is what I tried but its doesn't give correct output: df2['etl_flag'] = 'U' to_insert = df1[~df1.apply(lambda x: (df2['alias_cd'] == x['alias_cd']) & (df2['country_cd'] == x['country_cd']), axis=1)] to_insert['etl_flag'] = 'I' df2 = pd.concat([df2, to_insert], ignore_index=True) to_delete = df2[~df2.apply(lambda x: (df1['alias_cd'] == x['alias_cd']) & (df1['country_cd'] == x['country_cd']), axis=1)] to_delete['etl_flag'] = 'D' final_df = pd.concat([df2, to_delete], ignore_index=True) final_df.sort_values(by=['alias_cd', 'country_cd'], inplace=True) print(final_df[['alias_cd', 'country_cd', 'pos_name', 'ts_allocated', 'tr_id', 'ty_name', 'etl_flag']]) 

Answer: IIUC, you can first perform a left-merge with indicator on df2 to identify the U/D status, then concat. df1 always gets an I: # columns used as primary key cols = ['alias_cd', 'country_cd'] out = pd.concat( [df2.assign(etl_flag=df2.merge(df1[cols].drop_duplicates(), on=cols, how='left', indicator=True) ['_merge'].map({'left_only': 'D', 'both': 'U'}) .values), df1.assign(etl_flag='I')] )  Output:  alias_cd country_cd pos_name ts_allocated tr_id ty_name etl_flag 0 12345 AU st1 200 None E2E U 1 12345 AU3 st2 100 None E2E D 0 12345 AU st1 100 None E2E I 1 12345 AU Jh 100 None E2E I 2 12345 AU2 Jh 100 None E2E I