eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I store some configuration details across several class attributes and one main class which references each of them. E.g. class A: a = 1 class B: b = 2 def __init__(self): self.a_ = A() x = B()  I would like to display all class (and instance) attributes in a dictionary, i.e. {'b': 2, 'a_': {'a': 1}}  I understood __dict__ does not access class attributes. x.__dict__ # {'a_': <__main__.A at 0x1b888d9e530>} x.__dict__['a_'].__dict__ # {} -- why not {'a': 1}?  So I tried below class methods but the result is still not right:  @classmethod def to_dict(cls): return vars(cls) @classmethod def to_dict2(cls): return cls.__dict__ 

Answer: You can implement a Serializable class that both A and B inherit from that has a custom method to_dict() to achieve your desired output: class Serializable: def to_dict(self): d = {} for key, value in self.__class__.__dict__.items(): if not key.startswith('__') and not callable(value): d[key] = value for key, value in self.__dict__.items(): if hasattr(value, 'to_dict'): d[key] = value.to_dict() else: d[key] = value return d class A(Serializable): a = 1 class B(Serializable): b = 2 def __init__(self): self.a_ = A() x = B() print(x.to_dict())  Output for above: {'b': 2, 'a_': {'a': 1}}. Note this approach will have issues if your classes have circular references, non-serializable objects, attributes with leading underscores, or dynamically added attributes. But it works for your given contrived example!
Question: It turns out that the tag name should be: "ix:nonfraction" This does not work. No "xi" tag is found. from bs4 import BeautifulSoup text = """ <td style="BORDER-BOTTOM:0.75pt solid #7f7f7f;white-space:nowrap;vertical-align:bottom;text-align:right;">$ <ix:nonfraction name="ecd:AveragePrice" contextref="P01_01_2022To12_31_2022" unitref="Unit_USD" decimals="2" scale="0" format="ixt:num-dot-decimal">97.88</ix:nonfraction> </td> """ soup = BeautifulSoup(text, 'lxml') print(soup) ix_tags = soup.find_all('ix') print(ix_tags)  But the following works. I don't see a difference. Why is it? Thanks a lot! html_content = """ <html> <body> <ix>Tag 1</ix> <ix>Tag 2</ix> <ix>Tag 3</ix> <p>Not an ix tag</p> </body> </html> """ soup = BeautifulSoup(html_content, 'lxml') ix_tags = soup.find_all('ix') for tag in ix_tags: print(tag.text) 

Answer: The issue here arises from how BeautifulSoup handles namespaced tags like <ix:nonfraction>. With the lxml parser, namespaced tags might not be correctly parsed or recognized. In the XML you provided, ix is the namespace, and nonfraction is the local name of the element. In XML, a namespace is a method to avoid name conflicts by differentiating elements or attributes within XML documents. The ix:nonfraction tag indicates that the nonfraction element is part of the ix namespace. To correctly find namespaced tags like <ix:nonfraction> when using the lxml parser, you should use the exact tag name in your find_all call: ix_tags = soup.find_all('ix:nonfraction')  If you want to find the tags without providing the namespace, then you can use the xml parser which handles namespaced tags much more gracefully. soup = BeautifulSoup(text, 'xml') ix_tags = soup.find_all('nonfraction')  Sample run: from bs4 import BeautifulSoup text = """ <td style="BORDER-BOTTOM:0.75pt solid #7f7f7f;white-space:nowrap;vertical-align:bottom;text-align:right;">$ <ix:nonfraction name="ecd:AveragePrice" contextref="P01_01_2022To12_31_2022" unitref="Unit_USD" decimals="2" scale="0" format="ixt:num-dot-decimal">97.88</ix:nonfraction> </td> """ soup = BeautifulSoup(text, 'lxml') ix_tags = soup.find_all('ix:nonfraction') print(ix_tags) soup = BeautifulSoup(text, 'xml') ix_tags = soup.find_all('nonfraction') print(ix_tags)  Output: [<ix:nonfraction contextref="P01_01_2022To12_31_2022" decimals="2" format="ixt:num-dot-decimal" name="ecd:AveragePrice" scale="0" unitref="Unit_USD">97.88</ix:nonfraction>] [<nonfraction contextref="P01_01_2022To12_31_2022" decimals="2" format="ixt:num-dot-decimal" name="ecd:AveragePrice" scale="0" unitref="Unit_USD">97.88</nonfraction>]
Question: I want to get image file extensions from image URLs like below: from os.path import splitext image = ['ai','bmp','gif','ico','jpeg','jpg','png','ps','psd','svg','tif','tiff','webp'] def splitext_(path, extensions): for ext in extensions: if path.endswith(ext): return path[:-len(ext)], path[-len(ext):] return splitext(path) val = "https://dkstatics-public.digikala.com/digikala-products/9f4cb4e049e7a5d48c7bc22257b5031ee9a5eae8_1602179467.jpg?x-oss-process=image/resize,m_lfit,h_300,w_300/quality,q_80" #val = "https://www.needmode.com/wp-content/uploads/2023/04/%D9%84%D9%88%D8%A7%D8%B2%D9%85-%D8%AA%D8%AD%D8%B1%DB%8C%D8%B1.webp" ex_filename, ext = splitext_(val,image) ex_extension = ext.replace(".", "", 1) im_extension = ex_extension.lower() print(im_extension)  The problem is this method not working on URLs like below https://dkstatics-public.digikala.com/digikala-products/9f4cb4e049e7a5d48c7bc22257b5031ee9a5eae8_1602179467.jpg?x-oss-process=image/resize,m_lfit,h_300,w_300/quality,q_80  The result is nothing for the example image URL, but it's working on normal URLs.

Answer: Edit: here how to manage multiple extension. For this, it's better to use the @Andrej Kesely answer for parsing the url. Working on the url as string only will lead to have the host split and it's harder to manage (you would go to rewrite urlparse). from urllib.parse import urlparse val = "https://dkstatics-public.digikala.com/digikala-products/9f4cb4e049e7a5d48c7bc22257b5031ee9a5eae8_1602179467.tar.gz?x-oss-process=image/resize,m_lfit,h_300,w_300/quality,q_80" parsed_url = urlparse(val) extension = parsed_url.path.rsplit(".")[1:] print(extension)  First answer. Here how you can do: val = "https://dkstatics-public.digikala.com/digikala-products/9f4cb4e049e7a5d48c7bc22257b5031ee9a5eae8_1602179467.jpg?x-oss-process=image/resize,m_lfit,h_300,w_300/quality,q_80" print(val.split("?")[0].split(".")[-1])  You split first on the question mark and keeping the url part, not the parameters. And then you split on the dot and keep the last part which is the extension. It won't work with multiple extensions like tar.gz, you would only have gz.
Question: I have the following setup:  root logger to log everywhere my programm adds a new handler after startup via an callback (for example a database) with CallbackHandler named logger in my modules do not call into the root-logger  Online Python Compiler <- you need to have main.py selected to Run main.py in there main.py import logging import logging.config import MyLogger from MyApp import MyApp MyLogger.init() _logger = logging.getLogger() # root def main() : _logger.error( "main - root logger" ) app = MyApp() # setup app and attach CallbackHandler to root logger app.testLog() # call named logger - should call root logger & callback handler if __name__ == "__main__" : main()  MyLogger.py import logging from logging import LogRecord import logging.config import os from typing import Callable LOG_PATH = "./logs" LOGGING_CONFIG : dict = { "version" : 1 , 'formatters': { 'simple': { 'format': '%(name)s %(message)s' }, }, "handlers" : { "ConsoleHandler" : { "class" : "logging.StreamHandler" , "formatter" : "simple" , } , } , "root" : { "handlers" : [ "ConsoleHandler" , ] , "level" : "DEBUG" , } } def init() : os.makedirs( LOG_PATH , exist_ok = True ) logging.config.dictConfig( LOGGING_CONFIG ) class CallbackHandler( logging.Handler ) : def __init__( self , level = logging.DEBUG , callback : Callable = None ) : super().__init__( level ) self._callback = callback def emit( self , record : LogRecord ) : if self._callback is not None : self._callback( record.name + " | " + record.msg )  MyApp.py import logging from MyLogger import CallbackHandler _logger = logging.getLogger( __name__ ) class MyApp : def __init__( self ) : rootLogger = logging.getLogger() rootLogger.addHandler( CallbackHandler( callback = self.myCallback ) ) def myCallback( self , msg : str ) : print( "CALLBACK: " + msg ) def testLog( self ) : _logger.error( "MyApp.testLog() - named logger" )  The docs say, named loggers do not inherit the parents handlers. But they propagate their log messages to the parent/root logger - which has handlers attached. However they do not get called with a named logger. The Problem: CallbackHandler.emit() is not called (if I remove the __name__ in MyApp.py: logging.getLogger(), the root logger gets referenced and the Callback-Handler is called) How do I :  initialize the root logger later in my program attach a custom Handler to the root logger use named loggers in my program propagate the the logs from named loggers to the root logger such that the logs use the custom root-logger-handler 

Answer: How to fix: add the following line to your LOGGING_CONFIG dict: "disable_existing_loggers" : False,   The problem is that the child logger is created before the logging gets its configuration and the default behaviour when configuring the logging is to disable existing loggers. Link to the docs (it's the last item in that section).
Question: I'm exploring BeautifulSoup and aiming to retain only specific tags in an HTML file to create a new one. I can successfully achieve this with the following program. However, I believe there might be a more suitable and natural approach without the need to manually append the strings. from bs4 import BeautifulSoup #soup = BeautifulSoup(page.content, 'html.parser') with open('P:/Test.html', 'r') as f: contents = f.read() soup= BeautifulSoup(contents, 'html.parser') NewHTML = "<html><body>" NewHTML+="\n"+str(soup.find('title')) NewHTML+="\n"+str(soup.find('p', attrs={'class': 'm-b-0'})) NewHTML+="\n"+str(soup.find('div', attrs={'id' :'right-col'})) NewHTML+= "</body></html>" with open("output1.html", "w") as file: file.write(NewHTML) 

Answer: You can have a list of desired tags, iterate through them, and use Beautiful Soup's append method to selectively include corresponding elements in the new HTML structure. from bs4 import BeautifulSoup with open('Test.html', 'r') as f: contents = f.read() soup = BeautifulSoup(contents, 'html.parser') new_html = BeautifulSoup("<html><body></body></html>", 'html.parser') tags_to_keep = ['title', {'p': {'class': 'm-b-0'}}, {'div': {'id': 'right-col'}}] # Iterate through the tags to keep and append them to the new HTML for tag in tags_to_keep: # If the tag is a string, find it in the original HTML # and append it to the new HTML if isinstance(tag, str): new_html.body.append(soup.find(tag)) # If the tag is a dictionary, extract tag name and attributes, # then find them in the original HTML and append them to the new HTML elif isinstance(tag, dict): tag_name = list(tag.keys())[0] tag_attrs = tag[tag_name] new_html.body.append(soup.find(tag_name, attrs=tag_attrs)) with open("output1.html", "w") as file: file.write(str(new_html))  Assuming you have an HTML document like the one below (which would have been helpful to include for reproducibility's sake): <!DOCTYPE html> <head> <title>Test Page</title> </head> <body> <p class="m-b-0">Paragraph with class 'm-b-0'.</p> <div id="right-col"> <p>Paragraph inside the 'right-col' div.</p> </div> <p>Paragraph outside the targeted tags.</p> </body> </html>  the resulting output1.html will contain the following content: <html> <body> <title>Test Page</title> <p class="m-b-0">Paragraph with class 'm-b-0'.</p> <div id="right-col"> <p>Paragraph inside the 'right-col' div.</p> </div> </body> </html>
Question: I have been trying to convert html to pdf. I have tried a lot of tools but none of them work. Now I am using playwright, it is converting the Page to PDF but it only gets the first screen view. From that page the content from right is trimmed. import os import time import pathlib from playwright.sync_api import sync_playwright filePath = os.path.abspath("Lab6.html") fileUrl = pathlib.Path(filePath).as_uri() fileUrl = "file://C:/Users/PMYLS/Desktop/Code/ScribdPDF/Lab6.html" with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() page.goto(fileUrl) for i in range(5): #(The scroll is not working) page.mouse.wheel(0, 15000) time.sleep(2) page.wait_for_load_state('networkidle') page.emulate_media(media="screen") page.pdf(path="sales_report.pdf") browser.close()  Html View  PDF file after running script  I have tried almost every tool available on the internet. I also used selenium but same results. I thought it was due to page not loaded properly, I added wait and manually scrolled the whole page to load the content. All giving same results. The html I am converting https://drive.google.com/file/d/16jEq52iXtAMCg2FDt3VbQN0dCQmdTip_/view?usp=sharing

Answer: Here's a somewhat dirty solution that worked on my end. The sleep and scroll isn't great and can probably be improved, but I'll leave this as a starter and see if I have time to tighten it up later (feel free to do the same). from playwright.sync_api import sync_playwright # 1.37.0 from time import sleep with open("index.html") as f: html = f.read() with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() page.set_content(html) # focus inside the annoying border to enable scroll page.click(".document_container") for i in range(10): page.mouse.wheel(0, 2500) sleep(0.5) # strip out the annoying border that messes up PDF generation page.evaluate("""() => { const el = document.querySelector(".document_scroller"); el.parentElement.appendChild(el.querySelector(".document_container")); el.remove(); }""") page.emulate_media(media="screen") page.pdf(path="sales_report.pdf") browser.close()  Two tricks:  Clicking inside the border area enables scrolling, which appears necessary to get everything to load. Ripping out the annoying border allows the PDF generation to capture all pages. When the border is present, there's no scroll on the main body, only on the interior container, which the PDF capture doesn't seem to understand.
Question: I'm using Langchain 0.0.345. I cannot get a verbose output of what's going on under the hood using the LCEL approach to chain building. I have this code: from langchain.chat_models import ChatOpenAI from langchain.prompts import ChatPromptTemplate from langchain.schema.output_parser import StrOutputParser from langchain.globals import set_verbose set_verbose(True) prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}") model = ChatOpenAI() output_parser = StrOutputParser() chain = prompt | model | output_parser chain.invoke({"topic": "ice cream"})  According to the documentation using set_verbose is the way to have a verbose output showing intermediate steps, prompt builds etc. But the output of this script is just a string without any intermediate steps. Actually, the module langchain.globals does not appear even mentioned in the API documentation. I have also tried setting the verbose=True parameter in the model creation, but it also does not work. This used to work with the former approach building with classes and so. How is the recommended and current approach to have the output logged so you can understand what's going on? Thanks!

Answer: You can add a callback handler to the invoke method's configuration. Like this: from langchain.callbacks.tracers import ConsoleCallbackHandler # ...your code chain.invoke({"topic": "ice cream"}, config={'callbacks': [ConsoleCallbackHandler()]})  Code with change incorporated: from langchain.chat_models import ChatOpenAI from langchain.prompts import ChatPromptTemplate from langchain.schema.output_parser import StrOutputParser from langchain.callbacks.tracers import ConsoleCallbackHandler prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}") model = ChatOpenAI() output_parser = StrOutputParser() chain = prompt | model | output_parser chain.invoke({"topic": "ice cream"}, config={'callbacks': [ConsoleCallbackHandler()]})  The output isn't the same as the original "verbose mode", but this is the closest alternative. Alternatives For more targeted output or less "verbosity" Try attaching a callback handler to specific objects. For example: ChatOpenAI().with_config({'callbacks': [ConsoleCallbackHandler()]})  You can learn more about customizing callbacks here For high verbosity Global debug still works with LCEL: from langchain.globals import set_debug set_debug(True) # your code  For a GUI you can use weights and biases or langsmith
Question: I have a pandas dataframe df1 as shown below: It shows exports volume from A to B, B to A and A to C in three rows. Trade is possible in both directions.  df1.to_dict() returns  {'Country1': {0: 'A', 1: 'B', 2: 'A'}, 'Country2': {0: 'B', 1: 'A', 2: 'C'}, 'Value': {0: 3, 1: 5, 2: 3}}  I want a second dataframe df2 based on df1 which shows the net trade volume between countries. For example, A to C has a net trade volume of 3 units, and B to A has a net trade volume of 2 units (5-3). This needs to be reflected in the second dataframe as shown below:  How can I automate creating df2 based on df1? I have large number of countries, so I want to automate this process.

Answer: You could swap the names, merge and filter: val = (df[['Country1', 'Country2']] .merge(df.rename(columns={'Country1': 'Country2', 'Country2': 'Country1'}), how='left')['Value'] .rsub(df['Value'], fill_value=0) ) out = (df.assign(**{'Net Value': val}) .query('`Net Value` >= 0') .drop(columns='Value') )  Output:  Country1 Country2 Net Value 1 B A 2.0 2 A C 3.0
Question: I am learning pandas and python. I have this dataframe: dfsupport = pd.DataFrame({'Date': ['8/12/2020','8/12/2020','13/1/2020','24/5/2020','31/10/2020','11/7/2020','11/7/2020','4/4/2020','1/2/2020'], 'Category': ['Table','Chair','Cushion','Table','Chair','Mats','Mats','Large','Large'], 'Sales': ['1 table','3chairs','8 cushions','3Tables','12 Chairs','12Mats','4Mats','13 Chairs and 2 Tables', '3 mats, 2 cushions 4@chairs'], 'Paid': ['Yes','Yes','Yes','Yes','No','Yes','Yes','No','Yes'], 'Amount': ['93.78','$51.99','44.99','38.24','£29.99','29 21 only','18','312.8','63.77' ] })  which produces:  Date Category Sales Paid Amount 0 8/12/2020 Table 1 table Yes 93.78 1 8/12/2020 Chair 3chairs Yes 51.99 2 13/1/2020 Cushion 8 cushions Yes 44.99 3 24/5/2020 Table 3Tables Yes 38.24 4 31/10/2020 Chair 12 Chairs No 29.99 5 11/7/2020 Mats 12Mats Yes 29.21 6 11/7/2020 Mats 4Mats Yes 18 7 4/4/2020 Large 13 Chairs and 2 Tables No 312.8 8 1/2/2020 Large 3 mats, 2 cushions 4@chairs Yes 63.77  I want to find the date with the most sale, so I ran: print("######\n",dfsupport['Date'].value_counts().max())  which gives: 2  What I would now like to do is to unpack that 2 and find out which dates that was for and also which "Sales" occurred in each of those instances. I'm stuck and don't know how to print out those columns. Would appreciate some guidance.

Answer: Another possible solution, which uses pandas.DataFrame.groupby, pandas.DataFrame.transform and boolean indexing: s = dfsupport.groupby('Date')['Date'].transform(len) dfsupport[s.eq(s.max())]  Output:  Date Category Sales Paid Amount 0 8/12/2020 Table 1 table Yes 93.78 1 8/12/2020 Chair 3chairs Yes $51.99 5 11/7/2020 Mats 12Mats Yes 29 21 only 6 11/7/2020 Mats 4Mats Yes 18
Question: I have a problem where when I try to check if the given value's type is not what I expect I'll log it and raise an error. However, mypy is complaining. What I'm doing wrong? Simplified example: from __future__ import annotations from typing import Union from logging import getLogger class MyClass: def __init__(self, value: Union[float, int]) -> None: self.logger = getLogger("dummy") self.value = value def __add__(self, other: Union[MyClass, float, int]) -> MyClass: if not isinstance(other, (MyClass, float, int)): self.logger.error("Other must be either MyClass, float or int") # error: Statement is unreachable [unreachable] raise NotImplementedError return self.add(other) def add(self, other: Union[MyClass, float, int]) -> MyClass: if isinstance(other, MyClass): return MyClass(self.value + other.value) return MyClass(self.value + other)  Please notice it does not complain when I run it on mypy-play.net but locally it raises: main.py:13: error: Statement is unreachable [unreachable] Found 1 error in 1 file (checked 1 source file) 

Answer: Mypy is complaining because due to the input you set Union[MyClass, float, int] and your condition if not isinstance(other, (MyClass, float, int)):, if the arguments follows the given types, the code will never be reached. Mypy expect that everybody using your code will sent correct argument types (that's why you add types). You can disable this warning either with a type: ignore[unreachable] or just by unsettling the option in local. For the mypy-playground, you have to activate the option
Question: I updated my FastAPI to Pydantic 2.5.2 and I suddenly get the following warning in the logs. /usr/local/lib/python3.12/site-packages/pydantic/_migration.py:283: UserWarning: `pydantic.error_wrappers:ValidationError` has been moved to `pydantic:ValidationError`. warnings.warn(f'`{import_path}` has been moved to `{new_location}`.')  Is that problematic and do you know how to fix it?

Answer: Just use from from pydantic import ValidationError instead of from pydantic.error_wrappers import ValidationError. Now your code works correctly and it's just warning, but in the future versions of Pydantic it will cause the import error. If you don't import ValidationError in your code, it probably does one of the libraries you use. In this case just ignore this warning.
Question: I have been using plotly to create charts using OHLC data in a dataframe. The chart contains candlesticks on the top and volume bars at the bottom:  I want to annotate the candlestick chart (not the volume chart) but cannot work out how to do it. This code works to create the charts: # Plot chart # Create subplots and mention plot grid size fig = make_subplots(rows=2, cols=1, shared_xaxes=True, vertical_spacing=0.03, row_width=[0.2, 0.7]) # Plot OHLC on 1st row fig.add_trace(go.Candlestick(x=df.index, open=df["Open"], high=df["High"], low=df["Low"], close=df["Close"], name="OHLC"), row=1, col=1 ) # Bar trace for volumes on 2nd row without legend fig.add_trace(go.Bar(x=df.index, y=df['Volume'], showlegend=False), row=2, col=1) fig.update_layout(xaxis_rangeslider_visible=False, title_text=f'{ticker}') fig.write_html(fr"E:\Documents\PycharmProjects\xxxxxxxx.html")  And I tried adding the following after the candlestick add_trace but it doesn't work: fig.add_annotation(x=i, y=df["Close"], text="Test text", showarrow=True, arrowhead=1)  What am I doing wrong?

Answer: The issue is that you pass the whole df["Close"] series where you should pass only the value at index i, that is df.loc[i, "Close"]. This should work : fig.add_annotation(x=i, y=df.loc[i, "Close"], text="Test text", showarrow=True, arrowhead=1)
Question: Imagine I have a dataframe like this: import pandas as pd df = pd.DataFrame({"a":["","DATE","01-01-2012"], "b":["","ID",18], "c":["CLASS A","GOLF",3], "d":["","HOCKEY",4], "e":["","BASEBALL",2], "f":["CLASS B","GOLF",15], "g":["","HOCKEY",2], "h":["","BASEBALL",3] }) Out[33]: a b c d e f g h 0 CLASS A CLASS B 1 DATE ID GOLF HOCKEY BASEBALL GOLF HOCKEY BASEBALL 2 01-01-2012 18 3 4 2 15 2 3  I would like to add the strings in the first row to the names of those sports on the row below, but only before the beginning of the next "Class". Does anyone know how can I do that? So the result should be like this:  a b c ... f g h 0 CLASS A ... CLASS B 1 DATE ID CLASS A GOLF ... CLASS B GOLF CLASS B HOCKEY CLASS B BASEBALL 2 01-01-2012 18 3 ... 15 2 3  Later I will make the row 1 to be my header names, but this part I know how to do. I already tried to use df.iterrows but I got confused with the workflow.

Answer: Using replace+ffill to forward the CLASS, and a boolean mask to change the strings by boolean indexing: s = df.loc[0].replace('', np.nan).ffill() m = s.notna() df.loc[1, m] = s[m]+' '+df.loc[1, m]  Output:  a b c d e f g h 0 CLASS A CLASS B 1 DATE ID CLASS A GOLF CLASS A HOCKEY CLASS A BASEBALL CLASS B GOLF CLASS B HOCKEY CLASS B BASEBALL 2 01-01-2012 18 3 4 2 15 2 3  Side note: it might be better to have those two rows as a MultiIndex rather than rows with strings. This would enable you to benefit from vectorized operations on your numeric data.