eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am currently using the xml.etree Python library to parse HTML. After finding a target DOM element, I am attempting to extract its text. Unfortunately, it seems that the .text attribute is severely limited in its functionality and will only return the immediate inner text of an element (and not anything nested). Do I really have to loop through all the children of the ElementTree? Or is there a more elegant solution?

Answer: You can use itertext(), too. If you don’t like the whitespaces, indention and line break you can use strip(). import xml.etree.ElementTree as ET html = """<html> <head> <title>Example page</title> </head> <body> <p>Moved to <a href="http://example.org/">example.org</a> or <a href="http://example.com/">example.com</a>.</p> </body> </html>""" root = ET.fromstring(html) target_element = root.find(".//body") # get all text all_text = ''.join(target_element.itertext()) # get all text and remove line break etc. all_text_clear = ' '.join(all_text.split()) print(all_text) print(all_text_clear)  Output:  Moved to example.org or example.com. Moved to example.org or example.com.
Question: I have financial data where I need to save / find rows that have multiple same value and a condition where the same value happened more than / = 2 and not (value)equal to 0 or < 1. Say I have this:  A B C D E F G H I 5/7/2025 21:00 0 0 0 0 0 0 0 0 5/7/2025 21:15 0 0 19598.8 0 19598.8 0 0 0 5/7/2025 21:30 0 0 0 0 0 0 0 0 5/7/2025 21:45 0 0 0 19823.35 0 0 0 0 5/7/2025 22:00 0 0 0 0 0 0 0 0 5/7/2025 22:15 0 0 0 0 0 0 0 0 5/7/2025 22:30 0 0 0 19975.95 0 19975.95 0 19975.95 5/7/2025 23:45 0 0 0 0 0 0 0 0 5/8/2025 1:00 0 0 19830.2 0 0 0 0 0 5/8/2025 1:15 0 0 0 0 0 0 0 0 5/8/2025 1:30 0 0 0 0 0 0 0 0 5/8/2025 1:45 0 0 0 0 0 0 0 0  I want this along with other datas in those rows:  A B C D E F G H I 5/7/2025 21:15 0 0 19598.8 0 19598.8 0 0 0 5/7/2025 22:30 0 0 0 19975.95 0 19975.95 0 19975.95 

Answer: A simple approach could be to select the columns of interest, then identify if any value is duplicated within a row. Then select the matching rows with boolean indexing: mask = df.loc[:, 'B':].T out = df[mask.apply(lambda x: x.duplicated(keep=False)).where(mask >= 1).any()]  A potentially more efficient approach could be to use numpy. Select the values, mask the values below 1, sort them and identify if any 2 are identical in a row with diff + isclose: mask = df.loc[:, 'B':].where(lambda x: x>=1).values mask.sort() out = df[np.isclose(np.diff(mask), 0).any(axis=1)]  Output:  A B C D E F G H I 1 5/7/2025 21:15 0 0 19598.8 0.00 19598.8 0.00 0 0.00 6 5/7/2025 22:30 0 0 0.0 19975.95 0.0 19975.95 0 19975.95
Question: By nature, Playwright locator is blocking, so whenever it's trying to locate for an element X, it stops and waits until that element is located or it times out. However, I want to see if it is possible to make it so that it locates two elements at once, and, if either one is found, proceed forward, based on whichever was found. Is something like that possible in Python Playwright? Thanks

Answer: or_​ Added in: v1.33 Creates a locator matching all elements that match one or both of the two locators. Note that when both locators match something, the resulting locator will have multiple matches, potentially causing a locator strictness violation. Usage Consider a scenario where you'd like to click a "New email" button, but sometimes a security settings dialog appears instead. In this case, you can wait for either a "New email" button or a dialog and act accordingly. note If both "New email" button and security dialog appear on screen, the "or" locator will match both of them, possibly throwing the "strict mode violation" error. In this case, you can use locator.first to only match one of them. new_email = page.get_by_role("button", name="New") dialog = page.get_by_text("Confirm security settings") expect(new_email.or_(dialog).first).to_be_visible() if (dialog.is_visible()): page.get_by_role("button", name="Dismiss").click() new_email.click()
Question: I have a dataframe as below and I want remove the duplicates and want the output as mentioned below. Tried few things but not working as expected. New to pandas. import pandas as pd # Sample DataFrame data = { "some_id": "xxx", "some_email": "abc.xyz@somedomain.com", "This is Sample": [ { "a": "22", "b": "Y", "c": "33", "d": "x" }, { "a": "44", "b": "N", "c": "55", "d": "Y" }, { "a": "22", "b": "Y", "c": "33", "d": "x" }, { "a": "44", "b": "N", "c": "55", "d": "Y" }, { "a": "22", "b": "Y", "c": "33", "d": "x" }, { "a": "44", "b": "N", "c": "55", "d": "Y" } ] } df = pd.DataFrame(data) print(df)  The output is  some_id some_email This is Sample 0 xxx abc.xyz@somedomain.com {'a': '22', 'b': 'Y', 'c': '33', 'd': 'x'} 1 xxx abc.xyz@somedomain.com {'a': '44', 'b': 'N', 'c': '55', 'd': 'Y'} 2 xxx abc.xyz@somedomain.com {'a': '22', 'b': 'Y', 'c': '33', 'd': 'x'} 3 xxx abc.xyz@somedomain.com {'a': '44', 'b': 'N', 'c': '55', 'd': 'Y'} 4 xxx abc.xyz@somedomain.com {'a': '22', 'b': 'Y', 'c': '33', 'd': 'x'} 5 xxx abc.xyz@somedomain.com {'a': '44', 'b': 'N', 'c': '55', 'd': 'Y'}  I want to remove duplicates and the output should look like  some_id some_email This is Sample 0 xxx abc.xyz@somedomain.com {'a': '22', 'b': 'Y', 'c': '33', 'd': 'x'} 1 xxx abc.xyz@somedomain.com {'a': '44', 'b': 'N', 'c': '55', 'd': 'Y'}  How can this be achieved? I tried multiple ways some times it fails with unhashable dict. I have pretty big nested data frame like this. I am using pandas dataframe and python. New to this technology

Answer: The issue you're encountering (e.g., unhashable type: 'dict') happens because dictionaries are mutable and unhashable, so drop_duplicates() doesn't work directly on them. To deduplicate rows where one of the columns contains dictionaries, you can:  Convert dictionaries to strings, use drop_duplicates(), then  Convert the strings back to dictionaries (if needed).   Here’s a clean and simple way to achieve your desired output: https://code.livegap.com/?st=a50pbcrjkjk
Question: I have a file main_file.py that creates a global variable file_obj by opening a text file and imports a module imported_module.py which has functions that write to this file and therefore also has a global variable file_obj which I set equal to file_obj in main_file.py: main_file.py import imported_module as im file_obj = open('text_file.txt', mode='w') im.file_obj = file_obj def main(): a = 5 b = 7 im.add_func(a, b) im.multiply_func(a, b) return def add_func(x, y): z = x + y file_obj.write(str(z) + '\n') return main() file_obj.close()  imported_module.py file_obj = None def multiply_func(x, y): z = x * y file_obj.write(str(z) + '\n') return  If I close file_obj in main_file.py as above, does this also nicely close file_obj in imported_module.py? (In the MRE above, I could add im.file_obj.close() to main_file.py just to be sure. However, a generalization of this explicit approach does not appear possible if imported_module.py imports a second module imported_module0.py which also has a global variable file_obj and sets this variable to its own copy of file_obj with a command like im0.file_obj = file_obj.)

Answer: Yes. The two variables refer to the same file object. Closing either closes the object itself, it doesn't matter which variable you use to refer to it. This is no different from having two variable referring to the same list, a modification of one is visible through the other: a = [1, 2, 3] b = a a.append(4) print(b)  will print [1, 2, 3, 4]
Question: I’m having trouble working out why a widget doesn’t appear on my tkinter form. Here is what I’m doint:  Create a form Create a widget (a label) with the form as the master. Create a Notebook and Frame and add them to the form. Create additional widgets with the form as the master. Add the widgets to the form using grid, and specifying the in_ parameter.  Any widgets I create before the notebook and frame don’t appear, even though I don’t add them till after they’ve been created. Here is some sample code: form = tkinter.Tk() label1 = tkinter.ttk.Label(form, text='Test Label 1') # This one doesn’t appear notebook = tkinter.ttk.Notebook(form) notebook.pack(expand=True) mainframe = tkinter.ttk.Frame(notebook, padding='13 3 12 12') notebook.add(mainframe, text='Test Page') label2 = tkinter.ttk.Label(form, text='Test Label 2') # This one works entry = tkinter.ttk.Entry(form) label1.grid(in_=mainframe, row=1, column=1) label2.grid(in_=mainframe, row=2, column=1) entry.grid(in_=mainframe, row=3, column=1) form.mainloop()  Note that label doesn’t appear even though there is a space for it. If I print(id(form)) before and after creating the notebook and frame, they are the same, so it’s not as if the form itself has changed. Where has that first widget gone to and how can I get it to appear?

Answer: The behavior has to do with stacking order. Widgets created before the notebook are lower in the stacking order. In effect it is behind the notebook. You'll you correctly observed that a row has been allocated for the widget, but since it's behind the notebook it isn't visible. You can make it appear by calling lift on the widget to raise the stacking order: label1.lift()
Question: I'm working with a Polars DataFrame that contains a column with string values. I aim to replace specific values in this column using the str.replace_many() method. My dataframe: import polars as pl df = (pl.DataFrame({"Products": ["cell_foo","cell_fooFlex","cell_fooPro"]}))  Current approach: mapping= { "cell_foo" : "cell", "cell_fooFlex" : "cell", "cell_fooPro": "cell" } (df.with_columns(pl.col("Products").str.replace_many(mapping ).alias("Replaced")))  Output: shape: (3, 2) ┌──────────────┬──────────┐ │ Products ┆ Replaced │ │ --- ┆ --- │ │ str ┆ str │ ╞══════════════╪══════════╡ │ cell_foo ┆ cell │ │ cell_fooFlex ┆ cellFlex │ │ cell_fooPro ┆ cellPro │ └──────────────┴──────────┘  Desired Output: shape: (3, 2) ┌──────────────┬──────────┐ │ Products ┆ Replaced │ │ --- ┆ --- │ │ str ┆ str │ ╞══════════════╪══════════╡ │ cell_foo ┆ cell │ │ cell_fooFlex ┆ cell │ │ cell_fooPro ┆ cell │ └──────────────┴──────────┘  How can I modify my approach to ensure that replacements occur only when the entire string matches a key in the mapping?

Answer: The top-level Expr.replace() and .replace_strict() are for replacing entire "values". df.with_columns(pl.col("Products").replace(mapping).alias("Replaced"))  shape: (3, 2) ┌──────────────┬──────────┐ │ Products ┆ Replaced │ │ --- ┆ --- │ │ str ┆ str │ ╞══════════════╪══════════╡ │ cell_foo ┆ cell │ │ cell_fooFlex ┆ cell │ │ cell_fooPro ┆ cell │ └──────────────┴──────────┘
Question: I'm trying to get all the thing names that are part of a specific thing group in AWS IoT Core using a Python Lambda function. I checked the Boto3 documentation looking for a function that retrieves the names of things inside a specific thing group, but I couldn't find anything that does exactly that. Is there a way to fetch all the thing names from a thing group at once and store them in a list?

Answer: You can use the BOTO3 client to retrieve IoT things in a thing group. Here is the Python code. You need to use this Python code in an AWS Lambda function to address your use case. For additional AWS code examples, refer to the AWS Code Library -- where you will find thousands of examples in various SDKs, CLI, etc. import boto3 def list_things_in_group(group_name, region='us-east-1'): client = boto3.client('iot', region_name=region) try: response = client.list_things_in_thing_group( thingGroupName=group_name, recursive=False # Set to True if you want to include child groups ) things = response.get('things', []) if not things: print(f"No things found in group: {group_name}") else: print(f"Things in group '{group_name}':") for thing_name in things: print(f"- {thing_name}") describe_thing(client, thing_name) except client.exceptions.ResourceNotFoundException: print(f"Thing group '{group_name}' not found.") except Exception as e: print(f"Error: {e}") def describe_thing(client, thing_name): response = client.describe_thing(thingName=thing_name) print(f" Thing Name: {response.get('thingName')}") print(f" Thing ARN: {response.get('thingArn')}") print() # Example usage: if __name__ == "__main__": list_things_in_group("YourThingGroupName")
Question: I would like to understand how pattern matching works in Python. I know that I can match a value like so: >>> t = 12.0 >>> match t: ... case 13.0: ... print("13") ... case 12.0: ... print("12") ... 12  But I notice that when I use matching with a type like float(), it matches 12.0: >>> t = 12.0 >>> match t: ... case float(): ... print("13") ... case 12.0: ... print("12") ... 13  This seems strange, because float() evaluates to 0.0, but the results are different if that is substituted in: >>> t = 12.0 >>> match t: ... case 0.0: ... print("13") ... case 12.0: ... print("12") ... 12  I would expect that if 12.0 matches float(), it would also match 0.0. There are cases where I would like to match against types, so this result seems useful. But why does it happen? How does it work?

Answer: The thing that follows the case keyword is not an expression, but special syntax called a pattern. 0.0 is a literal pattern. It checks equality with 0.0. float() is a class pattern. It checks that the type is float. Since it is not an expression, it isn't evaluated and therefore is different from 0.0.
Question: So I have written this Python script to vote repeatedly (It's allowed) for a friend on a show at a local TV station. import os import time from pynput.keyboard import Key, Controller os.system("open -a Messages") time.sleep(3) keyboard = Controller() for i in range(50): keyboard.type("Example Message") print("Message typed") time.sleep(5) keyboard.press(Key.enter) print(f"======= {i+1} Message(s) Sent =======") time.sleep(40) print("Texting Complete")  During the first loop, everything works like it's supposed to, the program takes 5 seconds between typing and pressing Enter. However, in the loops thereafter, the pynput code seems to ignore time.sleep between keyboard.type & keyboard.press, running them immediately in succession, while the print statements still respect time.sleep in the terminal output. This isn't that big of an issue since it stills functions as intended most of the time, but about every 4th or 5th message gets sent before the program has finished typing, causing that vote not to get counted. I'm running the script in Visual Studio Code Version: 1.99.3 on a 2021 Macbook Pro with an M1 chip, if that matters. I have tried running the script unbuffered using the terminal, but it has made no difference. Any help would be appreciated.

Answer: Solved by user @furas in the comments here, keyboard.press() keeps the button pressed, so the code needed keyboard.release() to avoid the initial keyboard.press() being held in for the rest of the loops.
Question: Trying to get "Media created" timestamp and insert as the "Last modified date" with python for .mp4 and .m4a video, audio files (no EXIF). The "Media created" timestamp shows up and correctly in Windows with right click file inspection, but I can not get it with python. What am I doing wrong? (This is also a working fix for cloud storage changing the last modified date of files.)   enter from mutagen.mp4 import MP4, M4A from datetime import datetime import os def get_mp4_media_created_date(filepath): """ Extracts the "Media Created" date from an MP4 or M4A file. Args: filepath (str): The path to the MP4 or M4A file. Returns: datetime or None: The creation date as a datetime object, or None if not found. """ file_lower = filepath.lower() try: if file_lower.endswith(".mp4"): media = MP4(filepath) elif file_lower.endswith(".m4a"): media = M4A(filepath) else: return None # Not an MP4 or M4A file found_date = None date_tags_to_check = ['creation_time', 'com.apple.quicktime.creationdate'] for tag in date_tags_to_check: if tag in media: values = media[tag] if not isinstance(values, list): values = [values] for value in values: if isinstance(value, datetime): found_date = value break elif isinstance(value, str): try: found_date = datetime.fromisoformat(value.replace('Z', '+00:00')) break except ValueError: pass if found_date: break return found_date except Exception as e: print(f"Error processing {filepath}: {e}") return None if __name__ == "__main__": filepath = input("Enter the path to the MP4/M4A file: ") if os.path.exists(filepath): creation_date = get_mp4_media_created_date(filepath) if creation_date: print(f"Media Created Date: {creation_date}") else: print("Could not find Media Created Date.") else: print("File not found.") here 

Answer: As described here, the "Media created" value is not filesystem metadata. It's accessible in the API as a Windows Property. You can use os.utime to set "Media created" timestamp as the "Last modified date". Like import pytz import datetime import os from win32com.propsys import propsys, pscon file = 'path/to/your/file' properties = propsys.SHGetPropertyStoreFromParsingName(file) dt = properties.GetValue(pscon.PKEY_Media_DateEncoded).GetValue() if not isinstance(dt, datetime.datetime): # In Python 2, PyWin32 returns a custom time type instead of # using a datetime subclass. It has a Format method for strftime # style formatting, but let's just convert it to datetime: dt = datetime.datetime.fromtimestamp(int(dt)) dt = dt.replace(tzinfo=pytz.timezone('UTC')) print('Media created at', dt, dt.timestamp()) os.utime(file, (dt.timestamp(),dt.timestamp()))
Question: Given a dataframe, I want to apply a mapping with UDF but getting empty columns. data = [(1, 3), (2, 3), (3, 5), (4, 10), (5, 20)] df = spark.createDataFrame(data, ["int_1", "int_2"]) df.show() +-----+-----+ |int_1|int_2| +-----+-----+ | 1| 3| | 2| 3| | 3| 5| | 4| 10| | 5| 20| +-----+-----+  I have a mapping: def test_map(col): if col < 5: score = 'low' else: score = 'high' return score mapp = {} test_udf = F.udf(test_map, IntegerType())  I iterate here to populate mapp... for x in (1, 2): print(f'Now working {x}') mapp[f'limit_{x}'] = test_udf(F.col(f'int_{x}')) print(mapp) {'limit_1': Column<'test_map(int_1)'>, 'limit_2': Column<'test_map(int_2)'>} df.withColumns(mapp).show() +-----+-----+-------+-------+ |int_1|int_2|limit_1|limit_2| +-----+-----+-------+-------+ | 1| 3| NULL| NULL| | 2| 3| NULL| NULL| | 3| 5| NULL| NULL| | 4| 10| NULL| NULL| | 5| 20| NULL| NULL| +-----+-----+-------+-------+  The problem is I get null columns. What I'm expecting is: +-----+-----+-------+-------+ |int_1|int_2|limit_1|limit_2| +-----+-----+-------+-------+ | 1| 3| low | low | | 2| 3| low | low | | 3| 5| low | low | | 4| 10| low | high| | 5| 20| low | high| +-----+-----+-------+-------+  The reason I'm doing it is because I have to do for 100 columns. I heard that "withColumns" with a mapping is much faster than iterating over "withColumn" many times.

Answer: Your problem is that your UDF is registered to return an integer (defined to return an IntegerType()) while your Python function intends to return a string ("low" or "high"), so what you need to do is to set StringType() in your UDF return type: test_udf = F.udf(test_map, StringType())  Let me know if you want more explanation about UDFs!
Question: I have a long list of items that I want to assign a number to that increases by one every time the value in the list changes. Basically I want to categorize the values in the list. It can be assumed that the values in the list are always lumped together, but I don't know the number of instances it's repeating. The list is stored in a dataframe as of now, but the output needs to be a dataframe. Example: my_list = ['Apple', 'Apple', 'Orange', 'Orange','Orange','Banana'] grouping = pd.DataFrame(my_list, columns=['List'])  Expected output:  List Value 0 Apple 1 1 Apple 1 2 Orange 2 3 Orange 2 4 Orange 2 5 Banana 3  I have tried with a for loop, where it checks if the previous value is the same as the current value, but I imagine that there should be a nicer way of doing this.

Answer: Use pandas.factorize, and add 1 if you need the category numbers to start with 1 instead of 0: import pandas as pd my_list = ['Apple', 'Apple', 'Orange', 'Orange','Orange','Banana'] grouping = pd.DataFrame(my_list, columns=['List']) grouping['code'] = pd.factorize(grouping['List'])[0] + 1 print(grouping)  Output:  List code 0 Apple 1 1 Apple 1 2 Orange 2 3 Orange 2 4 Orange 2 5 Banana 3
Question: My matplotlib.__version__ is 3.10.1. I'm trying to underline some text and can not get it to work. As far as I can tell, Latex is installed and accessible in my system: import subprocess result = subprocess.run( ["pdflatex", "--version"], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) print(result.stdout)  results in: b'pdfTeX 3.141592653-2.6-1.40.25 (TeX Live 2023/Debian)\nkpathsea version 6.3.5\nCopyright 2023 Han The Thanh (pdfTeX) et al.\nThere is NO warranty. Redistribution of this software is\ncovered by the terms of both the pdfTeX copyright and\nthe Lesser GNU General Public License.\nFor more information about these matters, see the file\nnamed COPYING and the pdfTeX source.\nPrimary author of pdfTeX: Han The Thanh (pdfTeX) et al.\nCompiled with libpng 1.6.43; using libpng 1.6.43\nCompiled with zlib 1.3; using zlib 1.3\nCompiled with xpdf version 4.04\n'  Also the simple code: import matplotlib.pyplot as plt plt.text(0.5, 0.5, r'$\frac{a}{b}$') plt.show()  works as expected. Similar questions from 2012 (Underlining Text in Python/Matplotlib) and 2017 (matplotlib text underline) have accepted answers that fail with  RuntimeError: Failed to process string with tex because dvipng could not be found  A similar question from 2019 (Underlining not working in matplotlib graphs for the following code using tex) has no answer and it is my exact same issue, i.e.: import matplotlib.pyplot as plt plt.text(.5, .5, r'Some $\underline{underlined}$ text') plt.show()  fails with: ValueError: \underline{underlined} text ^ ParseFatalException: Unknown symbol: \underline, found '\' (at char 0), (line:1, col:1)  The 2017 question has a deleted answer that points to a closed PR in matplotlib's Github repo which points to another PR called Support \underline in Mathtext which is marked as a draft. Does my matplotlib version not support the underline Latex command?

Answer: As you have correctly found, \underline, is not a currently supported MathText command. But, matplotlib's MathText is not the same a LaTeX. To instead use LaTeX, you can do, e.g., import matplotlib.pyplot as plt # turn on use of LaTeX rather than MathText plt.rcParams["text.usetex"] = True plt.text(.5, .5, r'Some $\underline{underlined}$ text') plt.show()   You may have issues if your tex distribution does not ship with the type1cm package, in which can you may want to look at, e.g., https://stackoverflow.com/a/37218925/1862861.