eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Please clarify if the Annotated Transformer Encoder LayerNorm implementation is correct.  Transformer paper says the output of the sub layer is LayerNorm(x + Dropout(SubLayer(x))).  LayerNorm should be applied after the DropOut(SubLayer(x)) as per the paper:  However, the Annotated Transformer implementation does x + DropOut(SubLayer(LayerNorm(x))) where LayerNorm is applied before Sublayer, which is the other way around. class SublayerConnection(nn.Module): """ A residual connection followed by a layer norm. Note for code simplicity the norm is first as opposed to last. """ def __init__(self, size, dropout): super(SublayerConnection, self).__init__() self.norm = LayerNorm(size) self.dropout = nn.Dropout(dropout) def forward(self, x, sublayer): "Apply residual connection to any sublayer with the same size." return x + self.dropout(sublayer(self.norm(x))) # <--- LayerNorm before SubLayer 

Answer: Original paper applied Dropout to the Sub-Layer (Multi Head Attention) before Residual Connection and Layer Normalization. This is called Post Normalization.  dropout to the output of each sub-layer, before it is added to the sub-layer input (x) and (layer) normalized.  However, recent approach is Pre Normalization where LayerNorm is applied to the input x into the sub-layer as explained in Let's build GPT: from scratch, in code, spelled out.  Very few details about the Transformer have changed in the last five years, but there is something slightly departs from the original paper. You see that Add and Norm is applied after the transformation (Multi Head Attention). But now it is more common to apply LayerNorm before the transformation, so there is a reshuffling of the Layer Norm. This is called pre-norm formulation and that is the one we are going to implement as well.  This is proposed in On Layer Normalization in the Transformer Architecture.  The Annotated Transformer is also following this approach.
Question: I have an example class as follows: class MLP(nn.Module): # Declare a layer with model parameters. Here, we declare two fully # connected layers def __init__(self): # Call the constructor of the `MLP` parent class `Module` to perform # the necessary initialization. In this way, other function arguments # can also be specified during class instantiation, such as the model # parameters, `params` (to be described later) super().__init__() self.hidden = nn.Linear(20, 256) # Hidden layer self.out = nn.Linear(256, 10) # Output layer # Define the forward propagation of the model, that is, how to return the # required model output based on the input `X` def forward(self, X): # Note here we use the funtional version of ReLU defined in the # nn.functional module. return self.out(torch.relu(self.hidden(X)))  Calling the class is like this: net = MLP() net(X) Now, I need to create a similar class and function for a model with 4 layers: Layers Configuration Activation Function fully connected input size 128, output size 64 ReLU fully connected input size 64, output size 32 ReLU dropout probability 0.5 - fully connected input size 32, output size 1 Sigmoid I need to pass the following assertion: model = Net() assert model.fc1.in_features == 128 assert model.fc1.out_features == 64 assert model.fc2.in_features == 64 assert model.fc2.out_features == 32 assert model.fc3.in_features == 32 assert model.fc3.out_features == 1 x = torch.rand(2, 128) output = model.forward(x) assert output.shape == (2, 1), "Net() is wrong!"  Here is what I have so far: class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.fc1 = nn.Linear(128, 64) self.fc2 = nn.Linear(64, 32) self.dropout = nn.Dropout(p=0.5) self.fc3 = nn.Linear(32, 1) def forward(self, x): return self.fc3(torch.sigmoid(self.dropout(self.fc2(torch.relu(self.fc1(torch.relu(X)))))))  But I'm getting an error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x20 and 128x64)  How to resolve it?

Answer: You have a capital X in the forward method, while the function expects lowercase x. You must have a tensor with shape (2, 20) assigned to variable capital X.
Question: I'm using python-socketio and trying to pass the server instance to my app routers as dependency: main.py file: import socketio import uvicorn from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from src.api.rest.api_v1.route_builder import build_routes app = FastAPI() app.add_middleware( CORSMiddleware, allow_credentials=True, allow_methods=["*"], ) sio = socketio.AsyncServer(cors_allowed_origins="*", async_mode="asgi") app = build_routes(app) sio_asgi_app = socketio.ASGIApp(sio, app) @app.get("/") async def root(): return {"message": "Hello from main server !"} def start_server(): api_port = "0.0.0.0" api_host = "8000" uvicorn.run( "src.api.main:sio_asgi_app", host=api_host, port=int(api_port), log_level="info", reload=True, ) if __name__ == "__main__": start_server()  product_routes.py file: from typing import Annotated import socketio from fastapi import APIRouter, Depends router = APIRouter(prefix="/products") @router.get("/emit") async def emit_message( sio: Annotated[socketio.AsyncServer, Depends()] ) -> None: await sio.emit("reply", {"message": "Hello from server !"})  build_routes.py file: def build_routes(app: FastAPI) -> FastAPI: app.include_router( r_products, prefix="/v1", dependencies=[Depends(socketio.AsyncServer)], ) return app  So at this point I'm not sure how to pass that dependency to the routes. When I hit the emit endpoint I get this error response: { "detail": [ { "type": "missing", "loc": [ "query", "kwargs" ], "msg": "Field required", "input": null, "url": "https://errors.pydantic.dev/2.5/v/missing" } ] } 

Answer: You need to create function, where just return AsyncServer object. Then use this function as a dependency in your endpoint functions. It's better to create this dependency in separate file to avoid import errors. dependencies.py import socketio sio = socketio.AsyncServer(cors_allowed_origins="*", async_mode="asgi") def sio_dep() -> socketio.AsyncServer: return sio  then pass this function as a parameter to Depends: from typing import Annotated from dependencies import sio_dep from fastapi import APIRouter, Depends router = APIRouter(prefix="/products") @router.get("/emit") async def emit_message( sio: Annotated[socketio.AsyncServer, Depends(sio_dep)] ) -> None: await sio.emit("reply", {"message": "Hello from server !"})
Question: I have the following dataframe in polars. df = pl.DataFrame({ "Buy_Signal": [1, 0, 1, 0, 0], "Returns": np.random.normal(0, 0.1, 5), })  Ultimately, I want to do aggregations on column Returns conditional on different intervals - which are given by column Buy_Signal. In the above case the length is from each 1 to the end of the dataframe. What is the most polar way to do this? My "stupid" solution (before I can apply aggregations) is as follows: df = (df .with_columns(pl.col("Buy_Signal").cum_sum().alias("Holdings")) # (1) Determine returns for each holding period (>=1, >=2) and unpivot into one column .with_columns( pl.when(pl.col("Holdings") >= 1).then(pl.col("Returns")).alias("Port_1"), pl.when(pl.col("Holdings") >= 2).then(pl.col("Returns")).alias("Port_2"), ) )  The solution is obviously not working for many "buy signals". So I wrote a separate function calc_port_returns which includes a for loop which is then passed to polar's pipe function. See here: def calc_port_returns(_df: pl.DataFrame) -> pl.DataFrame: n_portfolios = _df["Buy_Signal"].sum() data = pl.DataFrame() _df = _df.with_columns(pl.col("Buy_Signal").cum_sum().alias("Holdings")) for i in range(1, n_portfolios + 1): tmp = ( _df.with_columns( pl.when(pl.col("Holdings") >= i).then(pl.col("Returns")).alias(f"Port_{i}"), ) .select(pl.col(f"Port_{i}")) ) data = pl.concat([data, tmp], how="horizontal") _df = pl.concat([_df, data], how="horizontal") return _df df = pl.DataFrame({ "Buy_Signal": [1, 0, 1, 0, 0], "Returns": np.random.normal(0, 0.1, 5), }) df.pipe(calc_port_returns)  What is the "polars" way to do this? In pandas I could imagine solving it using df.assign({f"Port_{i}": ... for i in range(1, ...)}) with a few prior extra columns / side calculations. Thanks for any suggestions.

Answer: It's pretty similar to what you were thinking... ( df .with_columns( Holdings = pl.col('Buy_Signal').cum_sum() ) .with_columns( pl.when(pl.col("Holdings")>=x) .then(pl.col("Returns")).alias(f"Port_{x}") for x in range(1,df['Buy_Signal'].sum()+1) ) ) shape: (5, 5) ┌────────────┬───────────┬──────────┬───────────┬───────────┐ │ Buy_Signal ┆ Returns ┆ Holdings ┆ Port_1 ┆ Port_2 │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ i64 ┆ f64 ┆ f64 │ ╞════════════╪═══════════╪══════════╪═══════════╪═══════════╡ │ 1 ┆ -0.05515 ┆ 1 ┆ -0.05515 ┆ null │ │ 0 ┆ 0.205705 ┆ 1 ┆ 0.205705 ┆ null │ │ 1 ┆ -0.068856 ┆ 2 ┆ -0.068856 ┆ -0.068856 │ │ 0 ┆ -0.141231 ┆ 2 ┆ -0.141231 ┆ -0.141231 │ │ 0 ┆ -0.028524 ┆ 2 ┆ -0.028524 ┆ -0.028524 │ └────────────┴───────────┴──────────┴───────────┴───────────┘
Question: I have created sklearn model for predicting ESG Scores. But for imputation reasons my input features matrix X were converted to np.ndarray. I found on apple developers website how to convert them, but there is optional kwargs input_features and output_features. import coremltools coreml_model = coremltools.converters.sklearn.convert(model, ["bedroom", "bath", "size"], "price") coreml_model.save('HousePricer.mlmodel')  I am wondering is it necessary match names of features in X with value I am sending to convert() method with input_features argument? I.e. imputer = KNNImputer() # Impute missing values in X X_imputed = imputer.fit_transform(X) X_imputed  Out: array([[ 5.00000e+00, 1.80000e+04, 3.27000e+08, ..., -2.20000e+07, 2.64000e+08, 2.64000e+08], [ 6.00000e+00, 1.32500e+05, 9.00000e+06, ..., 9.38000e+08, 1.90000e+07, 1.90000e+07], [ 2.00000e+00, 4.00000e+04, 2.37765e+08, ..., -1.88580e+07, 1.78696e+08, 1.78696e+08], ..., ])  So there are no labels now. The question is how to convert it to Core ML Model now, with no labels for input_features. Unfortunately I don't have access to macOS to try different approaches and test them. I would be grateful if you showed not only how to convert but also sample usage of the model with Swift. (Model based on Random Forest Regressor)

Answer: There is dedicated method called ct.converters.sklearn.convert for this goal. Here is a full example of the conversion: from sklearn.linear_model import LinearRegression import numpy as np import coremltools as ct # Load data X = np.random.rand(10,5).astype(np.ndarray) y = np.random.rand(10).astype(np.ndarray) # Train a model model = LinearRegression() model.fit(X, y) # save the scikit-learn model coreml_model = ct.converters.sklearn.convert(model) coreml_model.save('Predictor.mlmodel')  Printing coreml_model you get: In [13]: coreml_model Out[13]: input { name: "input" type { multiArrayType { shape: 5 dataType: DOUBLE } } } output { name: "prediction" type { doubleType { } } } predictedFeatureName: "prediction" metadata { userDefined { key: "com.github.apple.coremltools.source" value: "scikit-learn==1.1.1" } userDefined { key: "com.github.apple.coremltools.version" value: "7.1" } }
Question: I have to model a pydantic class from a JSON object which contains some invalid syntax keys. As an example: example = { "$type": "Menu", "name": "lunch", "children": [ {"$type": "Pasta", "title": "carbonara"}, {"$type": "Meat", "is_vegetable": false}, ] }  My pydantic classes at the moment looks like: class Pasta(BaseModel): title: str class Meat(BaseModel): is_vegetable: bool class Menu(BaseModel): name: str children: list[Pasta | Meat]  Now, this work except for $type field. If the field was called "dollar_type", I would simply create the following TranslationModel base class and let Pasta, Meat and Menu inherit from TranslationModel: class TranslationModel(BaseModel): @computed_field def dollar_type(self) -> str: return self.__class__.__name__  so that by executing Menu(**example).model_dump() I get { 'dollar_type': 'Menu', 'name': 'lunch', 'children': [ {'dollar_type': 'Pasta', 'title': 'carbonara'}, {'dollar_type': 'Meat', 'is_vegetable': False} ] }  But sadly I have to strictly follow the original json structure, so I have to use $type. I have tried using alias and model_validator by following the documentation but without success. How could I solve this? Thanks in advance

Answer: This very much looks like you would rather apply a discriminated union pattern. See the following example: from pydantic import BaseModel, Field from typing import Literal, Annotated example = { "$type": "Menu", "name": "lunch", "children": [ {"$type": "Pasta", "title": "carbonara"}, {"$type": "Meat", "is_vegetable": False}, ] } class Pasta(BaseModel): type: Literal["Pasta"] = Field("Pasta", alias="$type") title: str class Meat(BaseModel): type: Literal["Meat"] = Field("Meat", alias="$type") is_vegetable: bool AnyDish = Annotated[Pasta | Meat, Field(discriminator="type")] class Menu(BaseModel): name: str children: list[AnyDish] menu = Menu.model_validate(example) print(menu) menu.model_dump(by_alias=True)  Which prints: name='lunch' children=[Pasta(type='Pasta', title='carbonara'), Meat(type='Meat', is_vegetable=False)] {'name': 'lunch', 'children': [{'$type': 'Pasta', 'title': 'carbonara'}, {'$type': 'Meat', 'is_vegetable': False}]}  This pattern has multiple advantages:  It decouples the class name from the tag. This is usually preferred, because class names might change. But you might still want to read old files. The pattern is extensible and explicit. You can easily add new types of dishes later and just include them in the AnyDish type.  To reduce verbosity one can also introduce a short-cut like: class TypeLiteral: def __class_getitem__(cls, tag: str): return Annotated[Literal[tag], Field(default=tag, alias="$type")] class Pasta(BaseModel): type: TypeLiteral["Pasta"] title: str class Meat(BaseModel): type: TypeLiteral["Meat"] is_vegetable: bool  You can find more about discriminated unions in the pydantic docs: https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions I hope this is useful!
Question: I’m trying to write some of the columns of a pandas dataframe to an Excel file, but it’s taking too long. I’m currently using xlsxwriter to write all of the data to the file, which is much faster. How can I write only some of the columns to the file without sacrificing performance? My code for writing selected columns so far: for col_num, col_name in enumerate(zip(column_indexes, selected_columns)): for row_num in range(2, max_rows): value = df.iloc[row-2][col] if pd.isna(value): value ='' worksheet.write(row_num-1, col_num, value ) 

Answer: You can use the write_column method for col_num, col_name in enumerate(selected_columns): col_data = df[col_name].tolist() worksheet.write_column(1, col_num, col_data)   You can take a look and learn more about here: https://xlsxwriter.readthedocs.io/worksheet.html
Question: Let's say I have a type defined with TypeVar, like so: T = TypeVar('T') MyType = Union[list[T], tuple[T]] def func(a: MyType[int]): pass  I also want to have an optional version of it: MyTypeOpt = Optional[MyType] def opt_func(a: MyTypeOpt[int] = None): pass  But this is not to mypy's liking and I'm getting an error: Bad number of arguments for type alias, expected: 0, given: 1 [type-arg]. Is there a way to make it work without writing Optional[MyType[...]] every time?

Answer: I think you are missing the type parameter for your generic type MyType. mypy is also complaining about that. from typing import Optional, Sequence, Tuple, Union from typing_extensions import TypeVar T = TypeVar('T') MyType = Union[Sequence[T], Tuple[T]] MyTypeOpt = Optional[MyType[T]] # <--------- def func(a: MyType[int]) -> None: pass def opt_func(a: MyTypeOpt[int] = None) -> None: pass def opt_func2(a: Optional[MyType[int]] = None) -> None: pass
Question: I have two dataframes as below df1 = pd.DataFrame( { "names": ['alpha', 'bravo', 'charlie', 'delta', 'echo', 'foxtrot', 'golf'], "Debit": [0, 5000, 0, 5000, 3000, 0, 700], "Credit": [1000, 0, 2000, 0, 0, 8000, 0], } )  and df2 = pd.DataFrame( { "names": ['alpha', 'bravo', 'charlie', 'delta', 'echo', 'foxtrot'], "db_head": [1, 1, 1, 1, 1, 1], "cr_head": [2, 2, 2, 2, 2, 2], } )  The output I want is:  names Debit Credit head 0 alpha 0 1000 2 1 bravo 5000 0 1 2 charlie 0 2000 2 3 delta 5000 0 1 4 echo 3000 0 1 5 foxtrot 0 8000 2  I tried to merge but did not understand how to get the value from two of the last columns based on the current df's column values if merging without condition print(df1.merge(df2, how="inner", on="names"))  by simply merging both of the dataframes resulted like this  names Debit Credit db_head cr_head 0 alpha 0 1000 1 2 1 bravo 5000 0 1 2 2 charlie 0 2000 1 2 3 delta 5000 0 1 2 4 echo 3000 0 1 2 5 foxtrot 0 8000 1 2  tried these two methods but both of methods are giving error df1['acchead'] = [df2[df2['names'] == x].db_head.item() if y > 0 else df2[df2['names'] == x].cr_head.item() for x, y in [df1['names'], df1['Debit']]] df1['acchead'] = [df2[df2['names'] == x].db_head.item() if df1.Debit.item() > 0 else df2[df2['names'] == x].cr_head.item() for x in [df1['names']]]  Any Help would be appreciated.

Answer: You can merge, then post-process the output with pop and where: out = df1.merge(df2, on='names', how='inner') out['head'] = out.pop('db_head').where(out['Debit'].ne(0), out.pop('cr_head'))  Output:  names Debit Credit head 0 alpha 0 1000 2 1 bravo 5000 0 1 2 charlie 0 2000 2 3 delta 5000 0 1 4 echo 3000 0 1 5 foxtrot 0 8000 2  Other approach with a reshaping before the merge: (df1.assign(variable=np.where(df1['Debit'].ne(0), 'db_head', 'cr_head')) .merge(df2.melt('names', value_name='head'), on=['names', 'variable']) #.drop(columns='variable') )  Output:  names Debit Credit variable head 0 alpha 0 1000 cr_head 2 1 bravo 5000 0 db_head 1 2 charlie 0 2000 cr_head 2 3 delta 5000 0 db_head 1 4 echo 3000 0 db_head 1 5 foxtrot 0 8000 cr_head 2
Question: Delete 2 column headers and shift the whole column row to left in dataframe. Below is my dataframe.  trash1 trash2 name age 0 john 54 1 mike 34 2 suz 56 3 tin 78 4 yan 31  i need the final df as below  name age 0 john 54 1 mike 34 2 suz 56 3 tin 78 4 yan 31  I tried all the commands but its deleting the whole column. Please help me in this.

Answer: You can slice with iloc and rename with set_axis: N = 2 out = (df.iloc[:, :-N] .set_axis(df.columns[N:], axis=1) )  Output:  name age 0 john 54 1 mike 34 2 suz 56 3 tin 78 4 yan 31  Timing of answers for a generalized number of columns: Overall manually setting the name is reliably the fastest since the data is fully untouched, independently of the number of columns. The nice pop trick becomes very greedy after a threshold. Some of the approaches like shift+drop additionally cause the DataFrame to become fragmented
Question: The function is a part of the solution to the following problem: "Find all valid combinations of k numbers that sum up to n such that the following conditions are true: Only numbers 1 through 9 are used. Each number is used at most once. Return a list of all possible valid combinations. The list must not contain the same combination twice, and the combinations may be returned in any order." An example test case which fails with the code: import functools @functools.lru_cache(maxsize=None) def CombSum(k,n,num): if k<=0 or n<=0 or num>9: return [] if k==1 and n>=num and n<=9: return [[n]] ans=[] while num <= 9: for arr in CombSum(k-1,n-num,num+1): arr.append(num) ans.append(arr) num+=1 return ans print(CombSum(4,16,1))  produces [[9, 4, 2, 1], [8, 5, 2, 1], [7, 6, 2, 1], [8, 4, 3, 1], [7, 5, 3, 1], [6, 5, 4, 1, 5, 3, 2], [7, 4, 3, 2], [6, 5, 4, 1, 5, 3, 2]]  The above result is incorrect and when I remove the decorator below, everything works fine: import functools def CombSum(k,n,num): if k<=0 or n<=0 or num>9: return [] if k==1 and n>=num and n<=9: return [[n]] ans=[] while num <= 9: for arr in CombSum(k-1,n-num,num+1): arr.append(num) ans.append(arr) num+=1 return ans print(CombSum(4,16,1))  produces [[9, 4, 2, 1], [8, 5, 2, 1], [7, 6, 2, 1], [8, 4, 3, 1], [7, 5, 3, 1], [6, 5, 4, 1], [7, 4, 3, 2], [6, 5, 3, 2]]  This is the correct answer. Why is the decorator breaking the function?

Answer: You violated the decorator’s contract. You’re mutating elements of the returned list. The decorator should only be applied to Pure functions. You defined a Public API that returns a list of lists. Switch to returning an immutable tuple of tuples. Then your approach will be compatible with the LRU decorator.
Question: Given the code for a class definition, I am trying to extract all attributes and their comments ("" empty string if no comments). class Player(Schema): score = fields.Float() """ Total points from killing zombies and finding treasures """ name = fields.String() age = fields.Int() backpack = fields.Nested( PlayerBackpackInventoryItem, missing=[PlayerBackpackInventoryItem.from_name("knife")], ) """ Collection of items that a player can store in their backpack """  In the above example, we expected the parsed result to be: [ ("score", "Total points from killing zombies and finding treasures"), ("name", ""), ("age", ""), ("backpack", "Collection of items that a player can store in their backpack") ]  In my attempt below, it is failing to extract the comments properly, giving an output: [ ('score', 'Total points from killing zombies and finding treasures'), ('name', ''), ('age', ''), ('backpack', '') ]  How can the regex expression (or even the entire parsing logic) be fixed to handle the situations present in the example class code? Thanks import re code_block = '''class Player(Schema): score = fields.Float() """ Total points from killing zombies and finding treasures """ name = fields.String() age = fields.Int() backpack = fields.Nested( PlayerBackpackInventoryItem, missing=[PlayerBackpackInventoryItem.from_name("knife")], ) """ Collection of items that a player can store in their backpack """ ''' def parse_schema_comments(code): # Regular expression pattern to match field names and multiline comments pattern = r'(\w+)\s*=\s*fields\.\w+\([^\)]*\)(?:\n\s*"""\n(.*?)\n\s*""")?' # Find all matches using the pattern matches = re.findall(pattern, code, re.DOTALL) # Process the matches to format them as required result = [] for match in matches: field_name, comment = match comment = comment.strip() if comment else "" result.append((field_name, comment)) return result parsed_comments = parse_schema_comments(code_block) print(parsed_comments) 

Answer: I might use ast, e.g.: import ast code_block = '''class Player(Schema): score = fields.Float() """ Total points from killing zombies and finding treasures """ name = fields.String() age = fields.Int() backpack = fields.Nested( PlayerBackpackInventoryItem, missing=[PlayerBackpackInventoryItem.from_name("knife")], ) """ Collection of items that a player can store in their backpack """ ''' doc = {} for x in ast.parse(code_block).body[0].body: if isinstance(x, ast.Assign): name = x.targets[0].id doc[name] = '' else: doc[name] = x.value.value.strip() print(list(doc.items()))  Result (Attempt This Online!): [('score', 'Total points from killing zombies and finding treasures'), ('name', ''), ('age', ''), ('backpack', 'Collection of items that a player can store in their backpack')]