eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a way to display a PDF (only a single page, if that matters) in Python, without using the PyMuPDF library? I want to export my project via PyInstaller and including PyMuPDF increases the file size from ~40 to ~105 MB. Since I only want to display my pdf and don't need any of the advanced functionalities of PyMuPDF (or any editing/manipulation at all), I was wondering if there was a way to do so without so much overhead. The PDF is created during runtime with ReportLab.

Answer: I got it to work with the pypdfium2 package, which only adds ~3MB to the exported .exe instead of the ~60MB from mupdf. import pypdfium2 as pdfium image_buffer = io.BytesIO() doc2 = pdfium.PdfDocument(input=pdf_buffer, autoclose=True) doc2[0].render().to_pil().save(image_buffer, format='PNG') image_bytes = image_buffer.getvalue()  Here, both pdf_buffer and image_buffer are io.Bytes() for the pdf file input and the image file output respectively. However, setting this up for PyInstaller took bit: In the .spec file for the PyInstaller script, you need to manually add 3 files in order for pypdfium2 to properly work. In the a=Analysis(...) call you need to add these 3 lines to the datas argument: datas=[ (f'{site_packages_location}/pypdfium2_raw/pdfium.dll', 'pypdfium2_raw'), (f'{site_packages_location}/pypdfium2_raw/version.json', 'pypdfium2_raw'), (f'{site_packages_location}/pypdfium2/version.json', 'pypdfium2') ]  where site_packages_location = f"{os.getenv('LOCALAPPDATA')}/Programs/Python/Python312/Lib/site-packages/"  This answer has more info on that.
Question: I have a list of strings and a list of prefixes. I want to remove all elements from the list of strings that start with a prefix from the list of prefixes. I used a for loop, but why doesn't it seem to work? list_of_strings = ['test-1: foo', 'test-2: bar', 'test-3: cat'] list_of_prefixes = ['test1', 'test-2'] final_list = [] for i in list_of_strings: for j in list_of_prefixes: if not i.startswith(j): final_list.append(i) print(list(set(final_list)))  Currently the output is ['test-3: cat', 'test-1: foo', 'test-2: bar']  The output I want to get is final_list = ['test-3: cat'] 

Answer: Your approach doesn't work because you potentially perform an append for each element in list_of_prefixes, but if the string does start with one of the prefixes, it's guaranteed to not start with one of the others, so they all get added. With list comprehensions, generator expressions, and any, this is very straightforward. >>> list_of_strings = ['test-1: foo', 'test-2: bar', 'test-3: cat'] >>> list_of_prefixes = ['test1', 'test-2'] >>> filtered = [ ... s ... for s in list_of_strings ... if not any(s.startswith(p) for p in list_of_prefixes) ... ] >>> filtered ['test-1: foo', 'test-3: cat']  Note that 'test-1: foo' does not start with 'test1' or 'test-2'. If you meant for the list_of_prefixes to include 'test-1' then you would get the output you expect.
Question: I was working on this introduction to geospatial data analysis with Python. I've replicated each line of code in my own Jupyter notebook and have obtained the same results except for the last graph. The code for the graph is: fig, ax = plt.subplots(1, figsize=(20,20)) base = country[country['NAME'].isin(['Alaska','Hawaii']) == False].plot(ax=ax, color='#3B3C6E') florence.plot(ax=base, column='Wind', marker="<", markersize=10, cmap='cool', label="Wind speed(mph)") _ = ax.axis('off') plt.legend() ax.set_title("Hurricane Florence in US Map", fontsize=25) plt.savefig('Hurricane_footage.png',bbox_inches='tight')  and should yield the following graph :  However, when I copy that line of code in my notebook the legend is not working :  I thought it was my version of matplotlib so I updated it, but it's still not working. I don't see what else could be wrong.

Answer: They must be doing some classification by quantiles that wasn't covered in their tutorial : fig, ax = plt.subplots(1, figsize=(20, 20)) base = ( country[country["NAME"].isin(["Alaska", "Hawaii"]) == False].plot( ax=ax, color="#3B3C6E" ) ) florence.plot( ax=base, column="Wind", marker="<", markersize=10, cmap="cool", scheme="Quantiles", legend=True, legend_kwds={"title": "Wind speed(mph)"}, ) _ = ax.axis("off") ax.set_title("Hurricane Florence in US Map", fontsize=25) plt.savefig("Hurricane_footage.png", dpi=300, bbox_inches="tight")
Question: I'm using dependency_injector to manage DI. I don't understand how to release my resources using this library. I found shutdown_resources method but have no idea how to use it properly. Example: class Resource: """Resource example.""" def __init__(self): """.""" # Initialize session def close(self): """Release resources.""" # Close session class ApplicationContainer(DeclarativeContainer): """Application container.""" resource: Singleton[Resource] = Singleton[Resource](Resource) container = ApplicationContainer() # Do something container.shutdown_resources() # Call close method here 

Answer: It took some time but I found out. An initialization generator should be used for these purposes instead of a default constructor: from dependency_injector import containers, providers class Resource: """Resource example.""" def __init__(self): """.""" # Initialize session def close(self): """Release resources.""" # Close session def init_resource() -> None: resource = Resource() yield resource resource.close() class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" resource = providers.Resource(init_resource) container = ApplicationContainer() # Do something container.shutdown_resources() # Will call resource.close() here  UPD: I found even more clean and concise way to do it! You can inherit your resource from resources.Resource as mentioned here and override methods init and shutdown. This methods will be called on calling init_resources and shutdown_resources from container. from dependency_injector import resources, containers class Resource(resources.Resource): """Resource example.""" def init(self) -> Session: return create_session(...) def shutdown(self, session: Session) -> None: session.close() class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" resource: resources.Resource[Resource] = resources.Resource[Resource](Resource) if __name__ == "__main__": container = ApplicationContainer() container.init_resources() # init will be called here container.shutdown_resources() # shutdown will be called here
Question: I have a Python code of 78 lines using the following packages: import pandas as pd import base64 from bs4 import BeautifulSoup import os import win32com.client as win32 import pathlib  I ran the following commands: venv\Scripts\activate python -m pip install pandas python -m pip install pybase64 python -m pip install bs4 python -m pip install pywin32 python -m pip install pyinstaller pyinstaller --onefile to_HTML.py  I have created a virtual environment in which I installed only the above packages. Yet the EXE file created is 740Mb!! What am I doing wrong? How can I reduce it?

Answer: It doesn't look like you've installed pyinstaller within your virtual environment. I suspect it's attempting to use your global pyinstaller, which may be attempting to wrap any other packages you've installed globally. Try this: venv\Scripts\activate python -m pip install pyinstaller # install other dependencies as needed venv\Scripts\pyinstaller.exe --onefile to_HTML.py  Based on my quick test case, I get a very different file size when running with the virtual environment pyinstaller vs. the global one.
Question: I have a list created from a list(input()), that contains letter and numbers, like ['F', 'G', 'H', '1', '5', 'H'] I don't know the contents of the list before hand. How would I create a new list that shows the positions of strings that fit a predefined parameter so that I could receive an output like number_list = [3, 4] and letter_list = [0, 1, 2, 5]? I think my answer might involve index, enumerate(), and a for loop but I'm not sure how to go about filtering list contents in the way I want. I want the positional values of the strings in the list so I can use min() and max() functions so that the leftmost letter cant appear after the rightmost number. And later prevent the first number being used from being a 0.

Answer: You can use a simple for loop with isdigit(): def _collect(L): nums, alphas = [], [] for i, val in enumerate(L): if val.isdigit(): nums.append(i) else: alphas.append(i) return nums, alphas print(_collect(['F', 'G', 'H', '1', '5', 'H']))  Prints ([3, 4], [0, 1, 2, 5])
Question: I want to call my Python module from the Matlab. I received the error: Error using numpy_ops>init thinc.backends.numpy_ops  Python Error:  ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject.  The Python script is as follows import spacy def text_recognizer(model_path, text): try: # Load the trained model nlp = spacy.load(model_path) print("Model loaded successfully.") # Process the given text doc = nlp(text) ent_labels = [(ent.text, ent.label_) for ent in doc.ents] return ent_labels  The Matlab script is as follows % Set up the Python environment pe = pyenv; py.importlib.import_module('final_output'); % Add the directory containing the Python script to the Python path path_add = fileparts(which('final_output.py')); if count(py.sys.path, path_add) == 0 insert(py.sys.path, int64(0), path_add); end % Define model path and text to process model_path = 'D:\trained_model\\output\\model-best'; text = 'Roses are red'; % Call the Python function pyOut = py.final_output.text_recognizer(model_path, text); % Convert the output to a MATLAB cell array entity_labels = cell(pyOut); disp(entity_labels);  I found one solution to update Numpy, what I did, but nothing changed. I am using Python 3.9 and Numpy version 2.0.0 The error was received when I tried to call the Python module using a Matlab script. How can I fix the issue?

Answer: The reason is that pandas defines its numpy dependency freely as "anything newer than certain version of numpy". The problem occured, when numpy==2.0.0 has been released on June 16th 2024, because it is no longer compatible with your pandas version. The solution is to pin down the numpy version to any before the 2.0.0. Today it could be (this is the most recent numpy 1 release): numpy==1.26.4  To be added in your requirements or to the pip command you use (but together with installing pandas). Nowadays pip is very flexible and can handle the issue flawesly. You just need to ask it to install both pandas and numpy of given versions in the same pip install invocation.
Question: My main programming is done within Python, and want to invoke custom Powershell cmdlets I wrote. Added my .psm1 file to the $PSModulePath, and my cmdlets are always loaded. And I -NoProfile, and -NoLogo to invoke pwsh cmd a little bit faster. Something like cmd = ['pwsh', '-NoLogo', '-NoProfile', '-Command', cmdToRun] process = Popen(cmd, stderr=PIPE, stdout=PIPE)  But this is still taking 5+ secs to return/process. Does anyone know if there are other enhancements to run powershell scripts even faster? TIA

Answer: Python cannot host PowerShell in-process, so you cannot avoid the costly creation of a PowerShell child process, via pwsh, the PowerShell (Core) 7 CLI (the same applies analogously to powershell.exe, the Windows PowerShell CLI). -NoProfile would only make a difference if large / slowly executing $PROFILE file(s) were present, and -NoLogo is redundant, as it is implied by -Command. If you need to make PowerShell calls repeatedly in a given invocation of your application, you can mitigate the performance impact by launching a PowerShell CLI process in REPL mode, by passing -Command - and then feeding commands to it on demand via stdin, terminating each with two newlines (to ensure that the end of the command is recognized), similar to the approach in this post. You would then incur the penalty of the PowerShell child-process creation only once per run of your application, before submitting the first PowerShell command.
Question: I use apply to loop the rows and get the column names of feat1, feat2 or feat3 if they are equal to 1 and scored is equal to 0. The column names are then inserted into a new feature called reason. This solution doesn't scale to larger dataset. I'm looking for faster approach. How can I do that? df = pd.DataFrame({'ID':[1,2,3], 'feat1_tax':[1,0,0], 'feat2_move':[1,0,0], 'feat3_coffee': [0,1,0], 'scored':[0,0,1]}) def get_not_scored_reason(row): exclusions_list = [col for col in df.columns if col.startswith('feat')] reasons = [col for col in exclusions_list if row[col] == 1] return ', '.join(reasons) if reasons else None df['reason'] = df.apply(lambda row: get_not_scored_reason(row) if row['scored'] == 0 else None, axis=1) print(df)   ID feat1_tax feat2_move feat3_coffee scored reason 0 1 1 1 0 0 feat1_tax, feat2_move 1 2 0 0 1 0 feat3_coffee 2 3 0 0 0 1 None 

Answer: Another possible solution, around 8x faster than @Andrej Kesely's, according to rough estimates: feat_columns = df.filter(regex=r"^feat").columns reasons = df.mask(df["scored"] != 0, 0)[feat_columns].to_numpy() df["reason"] = np.array( [", ".join(feat_columns[row == 1]) if np.any(row == 1) else None for row in reasons] )   ID feat1_tax feat2_move feat3_coffee scored reason 0 1 1 1 0 0 feat1_tax, feat2_move 1 2 0 0 1 0 feat3_coffee 2 3 0 0 0 1 None
Question: Not sure if this is feasible or not. The implementation/example below is dummy, FYI. I have a Python class, Person. Each person has a public first name and a public last name attribute with corresponding private attributes - I'm using a descriptor pattern to manage access to the underlying private attributes. I am using the descriptor to count the number of times the attribute is accessed as well as obtain the underlying result. class AttributeAccessCounter: def __init__(self): self._access_count = 0 def __get__(self, instance, owner): self._access_count += 1 return getattr(instance, self.attrib_name) def __set_name__(self, obj, name): self.attrib_name = f'_{name}' @property def counter(self): return self._access_count class Person: first = AttributeAccessCounter() last = AttributeAccessCounter() def __init__(self, first, last): self._first = first self._last = last  From an instance of the class Person, how can I access the _access_count or property counter? john = Person('John','Smith') print(john.first) # 'John' print(john.first.counter) # AttributeError: 'str' object has no attribute 'counter' 

Answer: Currently you don't differentiate when the descriptor is accessed through the instance or through the class itself. property does this for example. It gives you the descriptor object when you access it through the class. You can do the same:  def __get__(self, instance, owner): if instance is None: return self self._access_count += 1 return getattr(instance, self.attrib_name)  Now the counter property of the descriptor class can be accessed. However there is another problem pointed out by @user2357112 in the comment. You store this count on descriptor and it's shared between different instances. You can't really tell first attribute of which instance of Person is accessed n times. To solve that, if you still want to store it in the descriptor object, one way is to use a dictionary and call e method for getting the count. Here is the complete code: from collections import defaultdict class AttributeAccessCounter: def __init__(self): self._access_counts = defaultdict(int) def __get__(self, instance, owner): if instance is None: return self self._access_counts[instance] += 1 return getattr(instance, self.attrib_name) def __set_name__(self, obj, name): self.attrib_name = f"_{name}" def count(self, obj): return self._access_counts[obj] class Person: first = AttributeAccessCounter() last = AttributeAccessCounter() def __init__(self, first, last): self._first = first self._last = last john = Person("John", "Smith") foo = Person("foo", "bar") print(Person.first.count(john)) # 0 print(john.first) # 'John' print(Person.first.count(john)) # 1 print(Person.first.count(foo)) # 0
Question: I created a 15 x 15 matrix in a pandas dataframe type. I want to make changes in some cells, following this logic:  The diagonal of the matrix is set to 0 In each row, if 0 appears in any position, the values in the next/ previous 5 columns should be updated to 1. (df.iloc[i][j] = 0 --> df.iloc[i][j+5] = 1 or df.iloc[i][j-5] = 1  Input (every cell is 0.9): row, col = 15, 16 a = pd.DataFrame.from_records([[0.9]*col]*row) a = a.loc[ : , a.columns != 0]   Expected output:  Below is my current script (which keeps running without an end): row, col = 15, 16 a = pd.DataFrame.from_records([[0.9]*col]*row) a = a.loc[ : , a.columns != 0] column3 = list(a) for i, j in a.iterrows(): for k in column3: if k == i + 1: a.iloc[i][k] = 0 for i, j in a.iterrows(): for k in column3: step = +5 if k <=5 else -5 while k <= len(a.index): if a.iloc[i][k] == 0: k += step a.iloc[i][k] = 1 pd.set_option('display.max_rows', None) pd.set_option('display.max_columns', None) a  Many thanks in advance for your help!

Answer: I would use numpy here and build a mask with roll: # distance to 0 N = 5 # replace diagonal with 0 np.fill_diagonal(a.values, 0) # build mask m = (a==0).to_numpy() # apply mask iteratively for i in range(1, a.shape[1]//N): a[np.roll(m, i*N, axis=1)] = 1  Variant using pandas' shift: N = 5 np.fill_diagonal(a.values, 0) m = (a==0) for i in range(1, a.shape[1]//N): a[m.shift(i*N, axis=1, fill_value=False)] = 1 a[m.shift(-i*N, axis=1, fill_value=False)] = 1  Output:  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 2 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 3 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 4 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 5 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 6 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 7 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 8 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 0.9 9 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 1.0 10 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 0.9 11 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 0.9 12 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 0.9 13 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0 0.9 14 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 1.0 0.9 0.9 0.9 0.9 0.0  Output as image for clarity:
Question: I have a data frame that looks like this :  I want to group the data frame by #PROD and #CURRENCY and replace TP with the contents of the Offshore data in the Loc column Without creating two data frames and joining them. The final output will look something like:  I was able to create the output by splitting the data frame into two (Onshore and Offshore ) and then creating a join on #PROD and #CURRENCY. However, I was wondering if there is a cleaner way to do this ? The Code for the Dataframe is : import pandas as pd data=[['Offshore','NY','A','USD','ABC_USD'],['Onshore','BH','A','USD',''], ['Onshore','AE','A','USD',''],\ ['Offshore','NY','A','GBP','GBP_ABC'],['Onshore','BH','A','GBP',''], ['Onshore','AE','A','GBP',''],\ ['Onshore','BH','A','EUR',''],['Onshore','AE','A','EUR','']] df = pd.DataFrame(data, columns=['Loc', 'Country','#PROD','#CURRENCY','TP']) df 

Answer: I think a merge is the most straightforward and efficient way to do this: df['TP'] = df[cols].merge(df[df['Loc'].eq('Offshore')], how='left')['TP'].values  No need to sort, no need to worry about which values are initially present. Alternatively: cols = ['#PROD', '#CURRENCY'] s = (df[cols].reset_index().merge(df[df['Loc'].eq('Offshore')]) .set_index('index')['TP'] ) df.loc[s.index, 'TP'] = s  Output:  Loc Country #PROD #CURRENCY TP 0 Offshore NY A USD ABC_USD 1 Onshore BH A USD ABC_USD 2 Onshore AE A USD ABC_USD 3 Offshore NY A GBP GBP_ABC 4 Onshore BH A GBP GBP_ABC 5 Onshore AE A GBP GBP_ABC 6 Onshore BH A EUR NaN 7 Onshore AE A EUR NaN
Question: I have a parent and two child classes defined like this import pygame class Person(): def __init__(self): self.image = pygame.image.load('person.png').convert_alpha() self.image = pygame.transform.scale(self.image, (int(self.image.get_width() * 0.5), int(self.image.get_height() * 0.5))) print('size: ', self.image.get_size()) class Teacher(Person): def __init__(self): super().__init__() self.image = pygame.image.load('teacher.png').convert_alpha() class Doctor(Person): def __init__(self): super().__init__() self.image = pygame.image.load('doctor.png').convert_alpha() self.image = pygame.transform.scale(self.image, (int(self.image.get_width() * 1.2), int(self.image.get_height() * 0.75))) ...  The size of the picture of person.png, teacher.png and doctor.png is 98x106, 125x173 and 97x178 respectively. When I run the following code, its output confused me. It seemed the code pygame.image.load() and pygame.transform.scale() in the child classes Teacher and Doctor didn't override the attribute defined in the parent class Person. pygame.display.set_mode((500, 500)) players = {'Teacher': Teacher(), 'Doctor': Doctor()} Output: pygame 2.4.0 (SDL 2.26.4, Python 3.10.9) Hello from the pygame community. https://www.pygame.org/contribute.html size: (49, 53) <---- expected to be (62, 86) size: (49, 53) <---- expected to be (116, 133)  What happened? What did I do wrong?

Answer: It's a matter of the order how everything is executed: In the subclasses, you call the super-constructor (super().__init__()), which still will load the picture person.png and output the size. Only afterwards the code in the constructors of the subclasses will be executed. So after the construction, the image property should be set correctly, but at the moment of the printout it is not. You could for example solve it, by having a file-name argument in the super constructor to which you hand the file names, fitting for the sub classes. Similarly, you should proceed with the scaling: class Person(): def __init__(self, filename='person.png', x_scale=0.5, y_scale=0.5): self.image = pygame.image.load(filename).convert_alpha() self.image = pygame.transform.scale(self.image, (int(self.image.get_width() * x_scale), int(self.image.get_height() * y_scale))) print('size: ', self.image.get_size()) class Teacher(Person): def __init__(self): super().__init__('teacher.png', 1.0, 1.0) class Doctor(Person): def __init__(self): super().__init__('doctor.png', 1.2, 0.75)