eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Using: import pandas as pd import numpy as np df = pd.DataFrame(np.random.randint(0,3,(100000,5000))) df = df.loc[:, (df != 0).any(axis=0)]  to get rid of columns containing only zeros is way too slow for a very large (1000000x2000) dataframe. Any suggestions how to speed this up? Thanks

Answer: There is a much faster way to implement that using Numba. Indeed, most of the Numpy implementation will create huge temporary arrays that are slow to fill and read. Moreover, Numpy will iterate over the full dataframe while this is often not needed (at least in your example). The point is that you can very quickly know if you need to keep a column by just iteratively check column values and early stop the computation of the current column if there is any 0 (typically at the beginning). Moreover, there is no need to always copy the entire dataframe (using about 1.9 GiB of memory): when all the columns are kept. Finally, you can perform the computation in parallel. However, there are performance-critical low-level catches. First, Numba cannot deal with Pandas dataframes, but the conversion to a Numpy array is almost free using df.values (the same thing applies for the creation of a new dataframe). Moreover, regarding the memory layout of the array, it could be better to iterate either over the lines or over the columns in the innermost loop.  This layout can be fetched by checking the strides of the input dataframe Numpy array. Note that the example use a row-major dataframe due to the (unusual) Numpy random initialization, but most dataframes tend to be column major. Here is an optimized implementation: import numba as nb @nb.njit('int_[:,:](int_[:,:])', parallel=True) def filterNullColumns(dfValues): n, m = dfValues.shape s0, s1 = dfValues.strides columnMajor = s0 < s1 toKeep = np.full(m, False, dtype=np.bool_) # Find the columns to keep # Only-optimized for column-major dataframes (quite complex otherwise) for colId in nb.prange(m): for rowId in range(n): if dfValues[rowId, colId] != 0: toKeep[colId] = True break # Optimization: no columns are discarded if np.all(toKeep): return dfValues # Create a new dataframe newColCount = np.sum(toKeep) res = np.empty((n,newColCount), dtype=dfValues.dtype) if columnMajor: newColId = 0 for colId in nb.prange(m): if toKeep[colId]: for rowId in range(n): res[rowId, newColId] = dfValues[rowId, colId] newColId += 1 else: for rowId in nb.prange(n): newColId = 0 for colId in range(m): res[rowId, newColId] = dfValues[rowId, colId] newColId += toKeep[colId] return res result = pd.DataFrame(filterNullColumns(df.values))  Here are the result on my 6-core machine: Reference: 1094 ms Valdi_Bo answer: 1262 ms This implementation: 0.056 ms (300 ms with discarded columns)  This, the implementation is about 20 000 times faster than the reference implementation on the provided example (no discarded column) and 4.2 times faster on more pathological cases (only one column discarded). If you want to reach even faster performance, then you can perform the computation in-place (dangerous, especially due to Pandas) or use smaller datatypes (like np.uint8 or np.int16) since the computation is mainly memory-bound.
Question: Why does the below code print error msg instead of ABC\nerror msg? class CustomException(Exception): """ABC""" def __init__(self, *args): super().__init__(*args) self.__str__ = self._wrapper(self.__str__) def _wrapper(self, f): def _inner(*args, **kwargs): return self.__doc__ + '\n' + f(*args, **kwargs) return _inner print(CustomException('error msg')) 

Answer: Operations backed by special methods usually explicitly look up the special method as a proper method not just as a callable attribute. Concretely, instead of self.__str__ the interpreter roughly looks at type(self).__str__.__get__(self, type(self)) – i.e. a descriptor __str__  on the class to be bound with the instance. To override a special method, it is thus necessary to override the class' descriptor instead of the instance' attribute. This can be done by a) declaring the special method as a slot, which handles the type(self).__str__ part, and b) assigning a function, which handles the __get__(self, type(self)) part. class CustomException(Exception): """ABC""" __slots__ = ("__str__",) # <<< magic def __init__(self, *args): super().__init__(*args) # vvv self.__str__ is the class' slot self.__str__ = self._wrapper(super().__str__) # AAA real __str__ lives on the super class def _wrapper(self, f): def _inner(*args, **kwargs): return self.__doc__ + '\n' + f(*args, **kwargs) return _inner print(CustomException('error msg'))  Note that since every instance behaves the same in this case, it is advisable to just define a new __str__ method in practice.
Question: I'd like to find the start and stop points of a column and flag them as shown below:     value flag     NaN NaN   NaN NaN   1 start   2 NaN   1 NaN   3 NaN   2 stop   NaN NaN   1 start   2 stop    

Answer:  start occurs when the current value is notnull and the previous value isnull stop occurs when the current value is notnull and the next value isnull  Generate these conditions using shift and assign using loc: start = df.value.notnull() & df.value.shift().isnull() stop = df.value.notnull() & df.value.shift(-1).isnull() df.loc[start, 'flag'] = 'start' df.loc[stop, 'flag'] = 'stop' # value flag # 0 NaN NaN # 1 NaN NaN # 2 1.0 start # 3 2.0 NaN # 4 1.0 NaN # 5 3.0 NaN # 6 2.0 stop # 7 NaN NaN # 8 1.0 start # 9 2.0 stop   Alternatively assign using mask: df['flag'] = df['flag'].mask(start, 'start') df['flag'] = df['flag'].mask(stop, 'stop')
Question: Is it possible to iterate dictionary in python using next(). (I need to access keys and values). Also would it work for extendable dictionaries with non-fixed length?

Answer: Use items to get the pairs key/value and iter() to be able to call next content = dict(zip(range(10), range(10, 20))) print(content) # {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, 8: 18, 9: 19} iterable = iter(content.items()) print(next(iterable)) # (0, 10) print(next(iterable)) # (1, 11) print(next(iterable)) # (2, 12) print(next(iterable)) # (3, 13)
Question: I can transform the target column to desired ordered numerical value using categorical encoding and ordinal encoding. But I am unable to perform inverse_transform as an error is showing which is written below. import pandas as pd import category_encoders as ce from sklearn.preprocessing import OrdinalEncoder lst = [ 'BRANCHING/ELONGATION', 'EARLY', 'EARLY', 'EARLY', 'EARLY', 'MID', 'MID', 'ADVANCED/TILLERING', 'FLOWERING', 'FLOWERING', 'FLOWERING', 'SEEDLING/EMERGED'] filtered_df = pd.DataFrame(lst, columns =['growth_state']) filtered_df['growth_state'].value_counts() EARLY 4 FLOWERING 3 MID 2 ADVANCED/TILLERING 1 SEEDLING/EMERGED 1 BRANCHING/ELONGATION 1 Name: growth_state, dtype: int64 dictionary = [{'col': 'growth_state', 'mapping':{'SEEDLING/EMERGED':0, 'EARLY':1, 'MID':2, 'ADVANCED/TILLERING':3, 'BRANCHING/ELONGATION':4, 'FLOWERING':5 }}] # instiating encoder encoder = ce.OrdinalEncoder(cols = 'growth_state', mapping= dictionary) filtered_df['growth_state'] = encoder.fit_transform(filtered_df['growth_state']) filtered_df growth_state 0 4 1 1 2 1 3 1 4 1 5 2 6 2 7 3 8 5 9 5 10 5 11 0  But when I perform inverse_transform: newCol = encoder.inverse_transform(filtered_df['growth_state']) AttributeError Traceback (most recent call last) <ipython-input-26-b6505b4be1e1> in <module> ----> 1 newCol = encoder.inverse_transform(filtered_df['growth_state']) d:\users\tiwariam\appdata\local\programs\python\python36\lib\site-packages\category_encoders\ordinal.py in inverse_transform(self, X_in) 266 for switch in self.mapping: 267 column_mapping = switch.get('mapping') --> 268 inverse = pd.Series(data=column_mapping.index, index=column_mapping.values) 269 X[switch.get('col')] = X[switch.get('col')].map(inverse).astype(switch.get('data_type')) 270 AttributeError: 'dict' object has no attribute 'index'  Note: the above column is a target column, I could have applied a label encoder as this is a classification-related problem. But I have adopted the above combination of categorical and ordinal encoding as variables are ordered in nature.

Answer: The error comes from this line in the inverse_transform source code: inverse = pd.Series(data=column_mapping.index, index=column_mapping.values)  It seems that even though the category_encoders documentation says that the mapping should be provided as a dictionary, their inverse_transform code is actually looking for a pd.Series: import pandas as pd from category_encoders import OrdinalEncoder df = pd.DataFrame({ 'growth_state': ['BRANCHING/ELONGATION', 'EARLY', 'EARLY', 'EARLY', 'EARLY', 'MID', 'MID', 'ADVANCED/TILLERING', 'FLOWERING', 'FLOWERING', 'FLOWERING', 'SEEDLING/EMERGED'] }) mapping = [{ 'col': 'growth_state', 'mapping': pd.Series(data={'SEEDLING/EMERGED': 0, 'EARLY': 1, 'MID': 2, 'ADVANCED/TILLERING': 3, 'BRANCHING/ELONGATION': 4, 'FLOWERING': 5}), 'data_type': object }] enc = OrdinalEncoder(cols=['growth_state'], mapping=mapping) df_transformed = enc.fit_transform(df) df_transformed.head() # growth_state # 0 4 # 1 1 # 2 1 # 3 1 # 4 1 df_inverse = enc.inverse_transform(df_transformed) df_inverse.head() # growth_state # 0 BRANCHING/ELONGATION # 1 EARLY # 2 EARLY # 3 EARLY # 4 EARLY
Question: I was wondering if it was possible to pass the results from the dependencies kwarg in include_router to the router that is passed to it. What I want to do is decode a JWT from the x-token header of a request and pass the decoded payload to the books routes. I know that I could just write authenticate_and_decode_JWT as a dependency of each of the routes in routers/book.py, but this would be quite repetitive for a large app. main.py from typing import Optional from jose import jwt from fastapi import FastAPI, Depends, Header, HTTPException, status from jose.exceptions import JWTError from routers import books app = FastAPI() def authenticate_and_decode_JWT(x_token: str = Header(None)): try: payload = jwt.decode(x_token.split(' ')[1], 'secret key', algorithms=['HS256']) return payload # pass decoded user information from here to books.router routes somehow except JWTError: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED) app.include_router( books.router, prefix="/books", dependencies=[Depends(authenticate_and_decode_JWT)], ) @app.get("/") def read_root(): return {"Hello": "World"}  routers/books.py from fastapi import APIRouter router = APIRouter() @router.get('/') def get_book(): # do stuff with decoded authenticated user data from JWT payload here pass @router.post('/') def create_book(): # do stuff with decoded authenticated user data from JWT payload here pass 

Answer: While the answer above about writing a middleware does work, if you don't want a middleware, you just want to use the include_router, then you can take the authenticate_and_decode_JWT method and extend it to write the JWT payload into the request object and then later have your routes read from that out from the request object. This way you only need to modify a few lines in your code for it to work: from fastapi import Request # <== new import def authenticate_and_decode_JWT(x_token: str = Header(None), request: Request): # <== request is a new param try: payload = jwt.decode(x_token.split(' ')[1], 'secret key', algorithms=['HS256']) request.token_payload = payload # type: ignore # <== store the token payload in request return payload except JWTError: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)  And then read from request: from fastapi import Request # <== new import @router.get('/', request: Request) # <== request is a new param def get_book(): # do stuff with decoded authenticated user data from JWT payload here print(request.token_payload) # <== access your token payload
Question: With Python 3.11.0a2+, and the following code: def my_fun(e): match e: case (1,): print("tuple (1,)") case [1]: print("list [1]") case _: print("I don't understand")  Calling the function with my_fun([1]) prints "tuple (1,)". Is this behavior correct? If I explicitly match against tuple((1, )) instead of (1,), it works as expected. If this is not a bug of the interpreter, what is the reason behind this seemingly weird behavior?

Answer: This is documented under Structural Pattern Matching  Like unpacking assignments, tuple and list patterns have exactly the same meaning and actually match arbitrary sequences. Technically, the subject must be a sequence. Therefore, an important exception is that patterns don’t match iterators. Also, to prevent a common mistake, sequence patterns don’t match strings.  and in PEP 635 -- Structural Pattern Matching: Motivation and Rationale  As in iterable unpacking, we do not distinguish between 'tuple' and 'list' notation. [a, b, c], (a, b, c) and a, b, c are all equivalent. While this means we have a redundant notation and checking specifically for lists or tuples requires more effort (e.g. case list([a, b, c])), we mimic iterable unpacking as much as possible.
Question: Hello I'm trying to read an excel file 'myFile.xlsx' using datatable.fread (version 1.0.0) function to speedup data manipulation. The problem is I had an AttributeError: module 'xlrd' has no attribute 'xlsx'. The command I used is: import datatable as dt DT = dt.fread("myFile.xlsx")  I checked the module where the error occurred is the module xls of datatable package: def read_xls_workbook(filename, subpath): try: import xlrd # Fixes the warning # "PendingDeprecationWarning: This method will be removed in future # versions. Use 'tree.iter()' or 'list(tree.iter())' instead." xlrd.xlsx.ensure_elementtree_imported(False, None) # Here xlrd.xlsx.Element_has_iter = True # and Here  Is there any solution to fix this issue? please.

Answer: The issue is that datatable package is not updated yet to make use of xlrd>1.2.0, so in order to make it work you have to install xlrd = 1.2.0 pip install xlrd==1.2.0  I hope it helped.
Question: I'm running Python code from an old repo that seems to no longer work but I can't figure out why or how to fix it. SETUP CODE (Google Colab Notebook) #@title Setup import pkg_resources print(pkg_resources.get_distribution("torch").version) from IPython.utils import io with io.capture_output() as captured: !git clone https://github.com/openai/CLIP # !pip install taming-transformers !git clone https://github.com/CompVis/taming-transformers.git !rm -Rf clipit !git clone https://github.com/mfrashad/clipit.git !pip install ftfy regex tqdm omegaconf pytorch-lightning !pip install kornia !pip install imageio-ffmpeg !pip install einops !pip install torch-optimizer !pip install easydict !pip install braceexpand !pip install git+https://github.com/pvigier/perlin-numpy # ClipDraw deps !pip install svgwrite !pip install svgpathtools !pip install cssutils !pip install numba !pip install torch-tools !pip install visdom !pip install gradio !git clone https://github.com/BachiLi/diffvg %cd diffvg # !ls !git submodule update --init --recursive !python setup.py install %cd .. !mkdir -p steps !mkdir -p models import sys sys.path.append("clipit")  THEN ERRORS ON SECOND TO LAST INE import clipit # To reset settings to default clipit.reset_settings() # You can use "|" to separate multiple prompts prompts = "underwater city" # You can trade off speed for quality: draft, normal, better, best quality = "normal" # Aspect ratio: widescreen, square aspect = "widescreen" # Add settings clipit.add_settings(prompts=prompts, quality=quality, aspect=aspect) # Apply these settings and run settings = clipit.apply_settings() clipit.do_init(settings) # generates error clipit.do_run(settings)   Working with z of shape (1, 256, 16, 16) = 65536 dimensions. loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth VQLPIPSWithDiscriminator running with hinge loss. Restored from models/vqgan_imagenet_f16_16384.ckpt --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) in () 12 # Apply these settings and run 13 settings = clipit.apply_settings() ---> 14 clipit.do_init(settings) 15 clipit.do_run(settings) 1 frames /usr/local/lib/python3.7/dist-packages/torch/jit/_script.py in fail(self, *args, **kwargs) 912 def _make_fail(name): 913 def fail(self, *args, **kwargs): --> 914 raise RuntimeError(name + " is not supported on ScriptModules") 915 916 return fail RuntimeError: requires_grad_ is not supported on ScriptModules 

Answer: For any unlucky soul that comes accross this issue this URL saved my soul https://github.com/mfrashad/text2art/issues/5
Question: To be more specific: To solve questions like How do I type hint a method with the type of the enclosing class? PEP 673 introduces typing.Self. The PEP is a Draft, but it currently an experimental type in typing_extensions 4.0.0 I tried using this in python 3.8 @dataclasses.dataclass class MenuItem: url: str title: str description: str = "" items: typing.List[typing_extensions.Self] = dataclasses.field(default_factory=list)  But it raises TypeError: Plain typing_extensions.Self is not valid as type argument  I could just use the literal string "MenuItem" instead. But I was wondering why this doesn't work.

Answer: Yes you can, but be aware of the uses of the package:  The typing_extensions module serves two related purposes:  Enable use of new type system features on older Python versions. For example, typing.TypeGuard is new in Python 3.10, but typing_extensions allows users on previous Python versions to use it too. Enable experimentation with new type system PEPs before they are accepted and added to the typing module.   This specific case was a bug in typing_extensions. It's being planned to get fixed in 4.0.1.
Question: I would like to import some waypoints/markers from a geojson file. Then determine the centroid of all of the points. My code calculates the centroid of each point not the centroid of all points in the series. How do I calculate the centroid of all points in the series? import geopandas filepath = r'Shiloh.json' gdf = geopandas.read_file(filepath) xyz = gdf['geometry'].to_crs('epsg:3587') print(type(xyz)) print(xyz) # xyz is a geometry containing POINT Z c = xyz.centroid # instead of calculating the centroid of the collection of points # centroid has calculated the centroid of each point. # i.e. basically the same X and Y data as the POINT Z.  The output from print(type(xyz)) and print(xyz) <class 'geopandas.geoseries.GeoSeries'> 0 POINT Z (2756810.617 248051.052 0.000) 1 POINT Z (2757659.756 247778.482 0.000) 2 POINT Z (2756907.786 248422.534 0.000) 3 POINT Z (2756265.710 248808.235 0.000) 4 POINT Z (2757719.694 248230.174 0.000) 5 POINT Z (2756260.291 249014.991 0.000) 6 POINT Z (2756274.410 249064.239 0.000) 7 POINT Z (2757586.742 248437.232 0.000) 8 POINT Z (2756404.511 249247.296 0.000) Name: geometry, dtype: geometry  the variable 'c' reports as (centroid of each point, not the centroid of the 9 POINT Z elements) : 0 POINT (2756810.617 248051.052) 1 POINT (2757659.756 247778.482) 2 POINT (2756907.786 248422.534) 3 POINT (2756265.710 248808.235) 4 POINT (2757719.694 248230.174) 5 POINT (2756260.291 249014.991) 6 POINT (2756274.410 249064.239) 7 POINT (2757586.742 248437.232) 8 POINT (2756404.511 249247.296) dtype: geometry 

Answer: first dissolve the GeoDataFrame to get a single shapely.geometry.MultiPoint object, then find the centroid: In [8]: xyz.dissolve().centroid Out[8]: 0 POINT (2756876.613 248561.582) dtype: geometry  From the geopandas docs:  dissolve() can be thought of as doing three things:  it dissolves all the geometries within a given group together into a single geometric feature (using the unary_union method), and it aggregates all the rows of data in a group using groupby.aggregate, and it combines those two results.   Note that if you have rows with duplicate geometries, a centroid calculated with this method will not appropriately weight the duplicates, as dissolve will first de-duplicate the records before calculating the centroid:  In [9]: gdf = gpd.GeoDataFrame({}, geometry=[ ...: shapely.geometry.Point(0, 0), ...: shapely.geometry.Point(1, 1), ...: shapely.geometry.Point(1, 1), ...: ]) In [10]: gdf.dissolve().centroid Out[10]: 0 POINT (0.50000 0.50000) dtype: geometry  To accurately calculate the centroid of a collection of points including duplicates, create a shapely.geometry.MultiPoint collection directly: In [11]: mp = shapely.geometry.MultiPoint(gdf.geometry) In [12]: mp.centroid.xy Out[12]: (array('d', [0.6666666666666666]), array('d', [0.6666666666666666]))
Question: I'm trying to sort the axis of my chart based on the values in the "sales" column. Once I apply fig.update_layout(yaxis={'categoryorder':'total descending'}) to my figure, the chart no longer removes irrelevant rows when filtering the dataframe by selecting the "099" value for the RadioItems callback option. code to recreate issue: import pandas as pd from plotly import tools import plotly.express as px import plotly.graph_objects as go import dash import dash_core_components as dcc import dash_html_components as html from jupyter_dash import JupyterDash import dash_bootstrap_components as dbc from dash.dependencies import Input, Output data = {'name': {0: 'G P', 1: 'D L', 2: 'T B', 3: 'N A', 4: 'P O', 5: 'Ho A'}, 'team': {0: '099', 1: '099', 2: '073', 3: '073', 4: '073', 5: '099'}, 'sales': {0: 88946, 1: 8123, 2: 6911, 3: 74796, 4: 8532, 5: -31289} } df1 = pd.DataFrame.from_dict(data) app = JupyterDash(external_stylesheets=[dbc.themes.SLATE]) template = 'plotly_dark' controls = dbc.Card( [ dbc.FormGroup( [ dbc.Label("Teams"), dcc.RadioItems( id='slm-radio', options=[{'label': 'All', 'value': 'All'}] + [{'label': k, 'value': k} for k in df1['team'].unique()], value='All', ), ], ) ] ) app.layout = dbc.Container( [ dbc.Row([ dbc.Col([controls],xs = 4), dbc.Col([ dbc.Row([ dbc.Col(dcc.Graph(id="sales_graph")), ]) ]), ]), html.Br(), ], fluid=True, ) @app.callback( Output("sales_graph", "figure"), [ Input("slm-radio", "value") ], ) def history_graph(team): dataset = df1 if not team == 'All': dataset = dataset[dataset['team']==team] else: dataset = dataset v_cat = dataset['name'] x_val = dataset['sales'] fig = go.Figure(go.Bar( x=x_val, y=v_cat, marker_color="#ff0000", orientation='h', width=0.25 )) fig.update_layout(template='plotly_dark') fig.update_layout(yaxis={'categoryorder':'total descending'}) return fig app.run_server(mode='inline', port = 8009)  The desired results is to remove the N A, P O, and T B columns when '099' is selected I get the desired results if:  the value for 'Ho A' is changed to positive sorting the axis is removed if graph is rendered using plotly locally without dash 

Answer: I was able to work around this issue by computing the ranking independently of the graphing step and placing the rankings into a series, rankings. I then passed the ranked series to fig.update_layout(yaxis={'categoryorder':'array', 'categoryarray':rankings['team_rank']})