eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've created a script in python to scrape certain fields from a webpage. When I use this link in the script, it produces all the data in json format and I can parse it accordingly. import requests link = 'https://api-emis.kemenag.go.id/v1/institutions/pontren/public/identity/K1hOenRreVRmaWYwSGVzcERWVFpjZz09' headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36' } with requests.Session() as s: s.headers.update(headers) res = s.get(link) container = res.json()['results'] account_bank_holder = container['account_bank_holder'] organizer_name = container['organizer_name'] print((account_bank_holder,organizer_name))  However, the problem is that I can't figure out where this portion K1hOenRreVRmaWYwSGVzcERWVFpjZz09 at the end of the link is coming from. This is how I found the link:  Navigate to this website Select 2023/2024 Genap Click the first item in the table: ACEH Then, click the first item in the new table: ACEH SELATAN Finally, click the first item in the new table: 500311010057  You should have the link by now, ending with the portion I mentioned above. Question: Where is the portion similar to this K1hOenRreVRmaWYwSGVzcERWVFpjZz09 at the end of the final link coming from?

Answer: So, developer has used aes encryption along with base64 encoding. import base64 from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import padding def aes_encrypt_cbc(plaintext: str) -> str: secret_key = b'a2c36eb2w1em50d6665dc5d61a68b400' base64_iv = "ZW1pc0Jhc2U2NElWa2V5cw==" iv = base64.b64decode(base64_iv) cipher = Cipher(algorithms.AES(secret_key), modes.CBC(iv), backend=default_backend()) encryptor = cipher.encryptor() padder = padding.PKCS7(128).padder() padded_plaintext = padder.update(plaintext.encode()) + padder.finalize() ciphertext = encryptor.update(padded_plaintext) + encryptor.finalize() return base64.b64encode(base64.b64encode(ciphertext).decode().encode()).decode() institution_id = "175974" encrypted_text = aes_encrypt_cbc(institution_id) print("Encrypted: ", encrypted_text)  Note: Initialization Vector and secret key may change in future.!
Question: I am using the pygame_gui library. I am trying to make a textbox that when pressed enter it will print the text inside the box to the console and reset it (so the textbox will be empty). It does indeed resets the textbox but for some reason the textbox doesnt get any new input until I click somewhere in the background and click again on the textbox and only then it shows the text it gets. The code: import pygame_gui, pygame pygame.init() screen = pygame.display.set_mode((500,500)) clock = pygame.time.Clock() manager = pygame_gui.UIManager((500,500)) manager.set_window_resolution((500,500)) text_box = pygame_gui.elements.UITextEntryLine(relative_rect=pygame.Rect(200,200,100,50), manager=manager, object_id='#text') run=True while run: UI_refresh = clock.tick(30)/1000 for event in pygame.event.get(): if event.type==pygame.QUIT: run=False elif event.type==pygame_gui.UI_TEXT_ENTRY_FINISHED and event.ui_object_id == '#text': print(text_box.text) text_box.set_text('') #here the text of the texbox is reset. manager.process_events(event) manager.update(UI_refresh) screen.fill('white') manager.draw_ui(screen) pygame.display.update()  I tried instead of reseting the textbox to set the text of it to some other value like 'A' (I changed the line text_box.set_text('') to text_box.set_text('A') ) but the textbox was still shown to be empty for some reason and I had the same problem. Plus I should mention that when setting the text it does show the changes for a bit less than a second before it disappears until I click somewhere on the screen and again on the textbox to show the changes.

Answer: I have gone through the source and I found out that if we add the line text_box.redraw() right after the set_text it fixes the problem, so the updated code is: import pygame_gui, pygame pygame.init() screen = pygame.display.set_mode((500,500)) clock = pygame.time.Clock() manager = pygame_gui.UIManager((500,500)) manager.set_window_resolution((500,500)) text_box = pygame_gui.elements.UITextEntryLine(relative_rect=pygame.Rect(200,200,100,50), manager=manager, object_id='#text') run=True while run: UI_refresh = clock.tick(30)/1000 for event in pygame.event.get(): if event.type==pygame.QUIT: run=False elif event.type==pygame_gui.UI_TEXT_ENTRY_FINISHED and event.ui_object_id == '#text': print(text_box.text) text_box.set_text('') #here the text of the texbox is reset. text_box.redraw() #this line fixes the problem. manager.process_events(event) manager.update(UI_refresh) screen.fill('white') manager.draw_ui(screen) pygame.display.update()
Question: What is the Python equivalent of the Perl ".." (range, or flip-flop) operator? for ( qw( foo bar barbar baz bazbaz bletch ) ) { print "$_\n" if /ar.a/ .. /az\w/; }  Output: barbar baz bazbaz  The Python workaround that I am aware of includes generator expression and indexing with the help of enumerate, but this seems cumbersome: import re lst = 'foo bar barbar baz bazbaz bletch'.split() idx_from = list(i for i, el in enumerate(lst) if re.search(r'ar.a', el))[0] idx_to = list(i for i, el in enumerate(lst) if re.search(r'az\w', el))[0] lst_subset = lst[ idx_from : (idx_to+1)] print(lst_subset) # ['barbar', 'baz', 'bazbaz']  Note: I am looking for just one range. There is currently no need to have multiple ranges.

Answer: When the operands aren't simple numbers, EXPR1 .. EXPR2 in scalar context is equivalent to following (except for the scope created by do { }): do { state $hidden_state = 0; if ( $hidden_state ) { ++$hidden_state; } else { $hidden_state = 1 if EXPR1; } my $rv = $hidden_state; # Or `$hidden_state > 1 && EXPR2` for `...`. if ( $hidden_state && EXPR2 ) { $rv .= "E0"; $hidden_state = 0; } $rv }  Since you only care whether the flip-flop returns true or false, the above simplifies to the following: do { state $hidden_state = false; $hidden_state ||= EXPR1; my $rv = $hidden_state; $hidden_state &&= EXPR2; $rv }  Now we have to translate this. Since the flip-flip is usually used as a generator, that's what I'll create. def flipflop( enumerable, start_cond, end_cond ): state = False for val in enumerable: if not state: state = start_cond( val ) if state: yield val if state: state = end_cond( val )  import re lst = 'foo bar barbar baz bazbaz bletch'.split() for x in flipflop( lst, lambda v: re.search( r'ar.a', v ), lambda v: re.search( r'az\w', v ) ): print( x )
Question: When I have numerical data, say index by some kind of time, it is straightforward to plot gaps in the data. For instance, if I have values at times 1, 2, 3, 5, 6, 7, I can set an np.nan at time 4 to break up the plot. import numpy as np import matplotlib.pyplot as plt x = [1, 2, 3, 4, 5, 6, 7] y = [10, 20, 30, np.nan, 10, 20, 30] plt.plot(x, y) plt.show() plt.close()   That sure beats the alternative of just skipping time 4! import numpy as np import matplotlib.pyplot as plt x = [1, 2, 3, 5, 6, 7] y = [10, 20, 30, 10, 20, 30] plt.plot(x, y) plt.show() plt.close()   However, I now have a y variable that is categorical. Mostly, the plotting is straightforward: just use the categories as the y. import numpy as np import matplotlib.pyplot as plt x = [1, 2, 3, 5, 6, 7] y = ["cat", "cat", "dog", "dog", "cat", "cat"] plt.plot(x, y) plt.show() plt.close()   This puts the categories on the y-axis, just as I want. However, when I do my np.nan trick to get the gap, I get a point plotted at np.nan. import numpy as np import matplotlib.pyplot as plt x = [1, 2, 3, 4, 5, 6, 7] y = ["cat", "cat", "dog", np.nan, "dog", "cat", "cat"] plt.plot(x, y) plt.show() plt.close()   How can I get my plots to go cat cat dog on 1, 2, 3, and then dog cat cat on 5, 6, 7, leaving a gap at 4?

Answer: To create a gap in a categorical plot handle np.nan differently asmatplotlib doesn’t natively interpret np.nan in categorical contexts (Gets converted to 'nan' string.). Solution: Splitting the data and then plot each segment separately. x = [1, 2, 3, 5, 6, 7] y = ["cat", "cat", "dog", "dog", "cat", "cat"] # Plot each segment separately to introduce a gap plt.plot(x[:3], y[:3]) plt.plot(x[3:], y[3:])   Alternative using Masks: x = [1, 2, 3, 4, 5, 6, 7] y = ["cat", "cat", "dog", "gap", "dog", "cat", "cat"] # Convert categorical data to numbers using a mapping categories = ["cat", "dog"] y_numeric = [categories.index(val) if val in categories else np.nan for val in y] y_masked = np.ma.masked_where(np.isnan(y_numeric), y_numeric) plt.plot(x, y_masked) plt.yticks(range(len(categories)), categories)   This allows gaps without splitting anything.
Question: I have pandas dataframe and I am trying to select multiple columns (column range starting from Test to Bio Ref). Selection has to start from column Test to any column whose name starts with Bio. Below is the sample dataframe. In reality it can contain:  any number of columns before Test column, any number of columns between Test & Bio Ref like 2,3,4,5 etc. any number of columns after Bio Ref. Bio Ref column can contain suffix in it but Bio Ref will be there as start of column name always.  df_chunk = pd.DataFrame({ 'Waste':[None,None], 'Test':['something', 'something'], '2':[None,None], '3':[None,None], 'Bio Ref':['2-50','15-100'], 'None':[None,None]}) df_chunk   Waste Test 2 3 Bio Ref None 0 None something None None 2-50 None 1 None something None None 15-100 None  I have tried below codes that work: df_chunk.columns.str.startswith('Bio') df_chunk[df_chunk.columns[pd.Series(df_chunk.columns).str.startswith('Bio')==1]]  Issue: But when I try to use them for multiple column Selection then it doesn't work: df_chunk.loc[:, 'Test':df_chunk.columns.str.startswith('Bio')] 

Answer: You can creates masks for boolean indexing: m1 = np.maximum.accumulate(df_chunk.columns=='Test') # array([False, True, True, True, True, True]) m2 = np.maximum.accumulate(df_chunk.columns.str.startswith('Bio')[::-1])[::-1] # array([ True, True, True, True, True, False]) # m1 & m2 # array([False, True, True, True, True, False]) out = df_chunk.loc[:, (m1&m2)]  Or identify the correct names to build a slice: start = 'Test' end = next(iter(df_chunk.columns[df_chunk.columns.str.startswith('Bio')]), None) out = df_chunk.loc[:, slice(start, end)]  Output:  Test 2 3 Bio Ref 0 something None None 2-50 1 something None None 15-100
Question: I use polars dataframes (new to this module) and I'm using some static typing, to keep my code tidy and clean for debugging purposes, and to allow auto-completion of methods and attributes on my editor. Everything goes well. However, when plotting things from dataframes with the altair API, as shown in the doc, I am unable to find the type of the returned object in polars. import polars as pl import typing as tp data = {"a": [0,0,0,0,1,1,1,2,2,3]} df = pl.DataFrame(data) def my_plot(df: pl.DataFrame, col: str) -> tp.Any: """Plot an histogram of the distribution of values of df[col]""" return df[col].value_counts( ).plot.bar( y="count", x=col ).properties( width=400, ) u = my_plot(df, "a") u.show()  How do I type the output of this function? The doc states the output of (...).plot is a DataFramePlot object but there is no info on this type, and anyway I'm using the output of (...).plot.bar(...) which has a different type. If I run type(u), I get altair.vegalite.v5.api.Chart but it seems sketchy to use this for static typing, and I don't want to import altair in my code as the altair methods I need are already included in polars. I couldn't find any info about this so any help is welcome Thanks!

Answer: You can do from __future__ import annotations from typing import TYPE_CHECKING if TYPE_CHECKING: import altair as alt def my_plot(df: pl.DataFrame, col: str) -> alt.Chart:
Question: I have a DataFrame where for each column I only want to show specific values based on the index, but these conditions are different for each column. This is what it looks like: data = {'a': [1,2,3,4,5], 'b': [10,20,30,40,50], 'c': [1,1,1,1,1]} df = pd.DataFrame(data) df: a b c 0 1 10 1 1 2 20 1 2 3 30 1 3 4 40 1 4 5 50 1  I now want to take values where index <3 for 'a', index <2 for 'b' and index = 4 for 'c'. Resulting in:  a b c 0 1.0 10.0 NaN 1 2.0 20.0 NaN 2 3.0 NaN NaN 4 NaN NaN 1.0  I tried the following: import pandas as pd df_a = df.loc[df.index<3, 'a'] df_b = df.loc[df.index<2, 'b'] df_c = df.loc[df.index==4, 'c'] df_result = pd.concat([df_a, df_b, df_c], axis=1)```  Which gives the desired result, but is there a more efficient way to do this? So if I would have a list for the "<" condition and a list for the "=" condition, then could I create the resulting filter in one go? It is fine if the NaNs become zeros, because that is what I want in the end anyway.

Answer: Another possible solution, whose steps are:  The np.stack function stacks these boolean conds, resulting in a 2D array, which is then transpose to align correctly the mask with the original df.  The [np.where]2 function then evaluates these conds, returning the element of df if True, and NaN otherwise.   conds = [df.index < 3, df.index < 2, df.index == 4] pd.DataFrame( np.where(np.stack(conds).T, df, np.nan), columns=df.columns).dropna(how='all')   Or, even simpler, using pandas where as @mozway suggests in a comment below (thanks!): conds = [df.index < 3, df.index < 2, df.index == 4] df.where(np.stack(conds).T).dropna(how='all')   Output:  a b c 0 1.0 10.0 NaN 1 2.0 20.0 NaN 2 3.0 NaN NaN 4 NaN NaN 1.0
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame({ 'a': [10, 20, 30, 50, 50, 50, 4, 100], 'b': [30, 3, 200, 25, 24, 31, 29, 2], 'd': list('aaabbbcc') })  Expected output:  a b d 0 10 30 a 1 20 3 a 2 30 200 a  The grouping is by column d. I want to return the groups that have at least two instances of this mask m = (df.b.gt(df.a))  This is what I have tried. It works but I wonder if there is a better/more efficient way to do it. out = df.groupby('d').filter(lambda x: len(x.loc[x.b.gt(x.a)]) >= 2) 

Answer: With pandas You could use a groupby.transform on the mask with sum to produce a boolean Series: m = df['b'].gt(df['a']) out = df[m.groupby(df['d']).transform('sum').ge(2)]  Output:  a b d 0 10 30 a 1 20 3 a 2 30 200 a  Intermediates:  a b d m transform('sum') ge(2) 0 10 30 a True 2 True 1 20 3 a False 2 True 2 30 200 a True 2 True 3 50 25 b False 0 False 4 50 24 b False 0 False 5 50 31 b False 0 False 6 4 29 c True 1 False 7 100 2 c False 1 False  Alternative: counts = m.groupby(df['d']).sum() out = df[df['d'].isin(counts.index[counts>=2])]  With numpy Alternatively, one could avoid the costly groupby with pure numpy. This first approach with add.reduceat requires the groups to be consecutive: groups = df['d'].ne(df['d'].shift()).values m = df['b'].gt(df['a']).values idx = np.nonzero(groups)[0] out = df[df['d'].isin(df['d'].iloc[idx[np.add.reduceat(m, idx)>=2]])]  This second one with pandas.factorize and numpy.bincount would work even with shuffled groups: a, idx = pd.factorize(df['d']) out = df[df['d'].isin(idx[np.bincount(a, weights=m) >= 2])]  Intermediates: ## reduceat approach groups = df['d'].ne(df['d'].shift()).values # array([ True, False, False, True, False, False, True, False]) m = df['b'].gt(df['a']).values # array([ True, False, True, False, False, False, True, False]) idx = np.nonzero(groups)[0] # array([0, 3, 6]) np.add.reduceat(m, idx)>=2 # array([ True, False, False]) idx[np.add.reduceat(m, idx)>=2] # array([0]) df['d'].iloc[idx[np.add.reduceat(m, idx)>=2]] # ['a'] df['d'].isin(df['d'].iloc[idx[np.add.reduceat(m, idx)>=2]]) # array([ True, True, True, False, False, False, False, False]) ## bincount approach a, idx = pd.factorize(df['d']) a # array([0, 0, 0, 1, 1, 1, 2, 2]) idx # Index(['a', 'b', 'c'], dtype='object') np.bincount(a, weights=m) # array([2., 0., 1.]) np.bincount(a, weights=m) >= 2 # array([ True, False, False]) idx[np.bincount(a, weights=m) >= 2] # Index(['a'], dtype='object') df['d'].isin(idx[np.bincount(a, weights=m) >= 2]) # array([ True, True, True, False, False, False, False, False])  Timings With groups of 3 rows (sorted members):  With groups of 3 rows (shuffled members; NB. excluding reduceat.):  With a fixed number of 20 groups (of about equal size) with consecutive members:  With a fixed number of 20 groups (of about equal size) with shuffled members (NB. excluding reduceat.):
Question: I have two data frames, df = pl.DataFrame({'url': ['https//abc.com', 'https//abcd.com', 'https//abcd.com/aaa', 'https//abc.com/abcd']})   conditions_df = pl.DataFrame({'url': ['https//abc.com', 'https//abcd.com', 'https//abcd.com/aaa', 'https//abc.com/aaa'], 'category': [['a'], ['b'], ['c'], ['d']]})  now I want a df, for assigning categories to the first df based on first match for the url starts with in second df, that is the output should be,    url category     https//abc.com ['a']   https//abcd.com ['b']   https//abcd.com/aaa ['b'] - this one starts with https//abcd.com, that is the first match   https//abc.com/abcd ['a'] - this one starts with https//abc.com, that is the first match    current code which works is like this, def add_category_column(df: pl.DataFrame, conditions_df) -> pl.DataFrame: # Initialize the category column with empty lists df = df.with_columns(pl.Series("category", [[] for _ in range(len(df))], dtype=pl.List(pl.String))) # Apply the conditions to populate the category column for row in conditions_df.iter_rows(): url_start, category = row df = df.with_columns( pl.when( (pl.col("url").str.starts_with(url_start)) & (pl.col("category").list.len() == 0) ) .then(pl.lit(category)) .otherwise(pl.col("category")) .alias("category") ) return df  but is there a way to achieve the same without using for loops, could we use join_where here, but in my attempts join_where does not work for starts_with

Answer: I would expect pl.DataFrame.join_where() to work, but apparently it doesn't allow pl.Expr.str.starts_with() condition yet - I get only 1 binary comparison allowed as join condition error. So you can use pl.DataFrame.join() and pl.DataFrame.filter() instead: ( df .join(conditions_df, how="cross") .filter(pl.col("url").str.starts_with(pl.col("url_right"))) .sort("url") .group_by("url", maintain_order=True) .agg(pl.col.category.first()) )  shape: (4, 2) ┌─────────────────────┬───────────┐ │ url ┆ category │ │ --- ┆ --- │ │ str ┆ list[str] │ ╞═════════════════════╪═══════════╡ │ https//abc.com ┆ ["a"] │ │ https//abc.com/abcd ┆ ["a"] │ │ https//abcd.com ┆ ["b"] │ │ https//abcd.com/aaa ┆ ["b"] │ └─────────────────────┴───────────┘  You can also use DuckDB integration with Polars and use lateral join: import duckdb duckdb.sql(""" select d.url, c.category from df as d, lateral ( select c.category from conditions_df as c where starts_with(d.url, c.url) limit 1 ) as c """)  ┌─────────────────────┬───────────┐ │ url │ category │ │ varchar │ varchar[] │ ├─────────────────────┼───────────┤ │ https//abc.com │ [a] │ │ https//abc.com/abcd │ [a] │ │ https//abcd.com/aaa │ [b] │ │ https//abcd.com │ [b] │ └─────────────────────┴───────────┘  however, you have to be careful cause in standard SQL specification row collections are unordered, so I'd not do that in production without adding explicit order by clause into lateral part.
Question: I'm using beautiful soup to extract some text content from HTML data. I have a div and several paragraph tags and the last paragraph is the copyright information with copyright logo , the year and some more info. the year is different based on what year the content was from , so i can't look for exact text but rest is always the same besides a variable year . is there a way i can delete/ignore the last paragraph? from bs4 import BeautifulSoup text_content = '<div><p>here is the header information </p><p> some text content </p> <p> another block of text</p> .....<p> 2024 copyright , all rights reserved </p>' bs = BeautifulSoup(text_content, "html.parser") only_text = " ".join([p.text for p in soup.find_all("p")])  I used beautiful soup to get all the text content , now i want to remove a particular paragraph.

Answer: Just store elements in a list and then pop the last item. soup = BeautifulSoup(text_content, "html.parser") # Find all the paragraph tags paragraphs = soup.find_all("p") # Check if the last paragraph contains a match if "copyright" in paragraphs[-1].text.lower(): # Remove the last paragraph paragraphs.pop() # Join the remaining paragraphs only_text = " ".join([p.text for p in paragraphs])  paragraphs = soup.find_all("p") - finds all the <p> tags and stores them in a list paragraphs.pop() - removes the last element from the list
Question: The loop runs over some number of models (n_mod) and is distributed among the n_cpus. As you will note, running this code as mpirun -np 4 python test_mpi.py produces 4 progress bars. This is understandable. But is there a way to use tdqm to get one progress bar which tells me how many models have been completed? from tqdm import tqdm from mpi4py import MPI import time comm = MPI.COMM_WORLD cpu_ind = comm.Get_rank() n_cpu = comm.Get_size() n_mod=100 for i in tqdm(range(n_mod)): if (cpu_ind == int(i/int(n_mod/n_cpu))%n_cpu): #some task here which is a function of i time.sleep(0.02) 

Answer: This can be achieved with a main node and worker node(s) setup. Essentially only the rank == 0 node will be updating a progress bar whilst the worker nodes will simply be informing the main node that they have completed the task. Worker: (Defined similarly to your above code) def worker(n_mod, size, rank): comm = MPI.COMM_WORLD step = size - 1 # Number of worker processes start = rank - 1 for i in range(start, n_mod, step): time.sleep(0.02) # Work # Notify the master that a task is done comm.send('done', dest=0, tag=1)  Main Node: def pbar_node(n_mod, size): comm = MPI.COMM_WORLD pbar = tqdm(total=n_mod, desc="Processing Models") completed = 0 while completed < n_mod: # Receive a message from any worker status = MPI.Status() comm.recv(source=MPI.ANY_SOURCE, tag=1, status=status) completed += 1 pbar.update(1) pbar.close()  I have tested this with mpirun -np 4 python test.py and get a single progress bar for 4 models sharing 100 tasks as shown in the screenshot.  Ensure that you take into account the fact there will be the number of workers plus a main node when dealing with distributing tasks so you do not get off-by-one errors. You can use these like so: comm = MPI.COMM_WORLD rank = comm.Get_rank() size = comm.Get_size() n_mod = N_TASKS if rank == 0: # pbar process pbar_node(n_mod)else: # Worker processes worker(n_mod, size, rank)
Question: I'm running ansible playbooks in python venv My playbooks often involve a mix of cloud infrastructure (AWS) and system engineering. I have configured them to run cloud infrastructure tasks with connection: local - this is to minimize access rights required on the target system. However since using venv I have a conflict in regards to the ansible_python_interpreter location:  on the target system they tend to be in a "default" location /usr/bin/python3 - I am not 100% sure if this is hard coded in ansible, or stored in a PATH variable on my local system I assume they are defined by  home = /opt/homebrew/opt/python@3.12/bin include-system-site-packages = false version = 3.12.5 executable = /opt/homebrew/Cellar/python@3.12/3.12.5/Frameworks/Python.framework/Versions/3.12/bin/python3.12 command = /opt/homebrew/opt/python@3.12/bin/python3.12 -m venv /Users/jd/projects/mgr2/ansible  Because of this, I cannot run a mixed playbook, either I need add  vars: ansible_python_interpreter: /Users/jd/projects/mgr2/ansible/bin/python3  to my playbook to run local tasks or remove this line to run target system tasks. I'm looking for a way to have python3 in the PATH variable, depending on which venv I am sourcing.

Answer: I had a similar problem, and I resolved it with interpreter fallback which can use a list of locations (attempted in order) unlike the interpreter config which is just a single path. In your inventory set this variable: ... ... ansible_python_interpreter_fallback: - /Users/jd/projects/mgr2/ansible/bin/python3 - /usr/bin/python3  Note that you can not do this in the config file, unfortunately, it has to be set in inventory (or by inventory plugin).