eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: everyone. I hope to train a CV model in the timm library on my dataset. Due to the shape of the input data is (batch_size, 15, 224, 224), I need to modify the "in_channel" of the first CNN layer of different CV models. I try different methods but still fail. Could you help me solve this problem? Thanks! import torch import torch.nn as nn import timm class FrequencyModel(nn.Module): def __init__( self, in_channels = 6, output = 9, model_name = 'resnet200d', pretrained = False ): super(FrequencyModel, self).__init__() self.in_channels = in_channels self.output = output self.model_name = model_name self.pretrained = pretrained self.m = timm.create_model(self.model_name, pretrained=self.pretrained, num_classes=output) for layer in self.m.modules(): if(isinstance(layer,nn.Conv2d)): layer.in_channels = self.in_channels break def forward(self,x): out=self.m(x) return out if __name__ == "__main__": x = torch.randn((8, 15, 224, 224)) model=FrequencyModel( in_channels = 15, output = 9, model_name = 'resnet200d', pretrained = False ) print(model) print(model(x).shape)  The error is:  RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[8, 15, 224, 224] to have 3 channels, but got 15 channels instead  I hope I can test different CV model easily but not adjust it one by one.

Answer: Do you want to change the first conv2d layer of resnet200d? Let me point out a few things that went wrong here.  Changing only the value of in_channels does not change the shape of the weight. resnet200d is composed of sequential layers and has a conv2d layer in them. So, you cannot access conv2d with a for statement like the code above. Use the apply method for a recursive approach. If you want to actually change the layer, use ._modules['module name']. If you access a layer with m.modules(), the layer does not change because deepcopy occurs. Thus, get both name and module using model.named_modules()  So you probably want to change it like this: import torch import torch.nn as nn import timm x = torch.randn((8, 15, 224, 224)) m = timm.create_model('resnet200d', pretrained=False, num_classes=9) m._modules['conv1']._modules['0'] = nn.Conv2d(15, 32, 3, stride=2, padding=1, bias=False) print(m) print(model(x).shape)  More generally, you can change like this: change_first_layer function that changes the in_channels of the first conv2d layer to 15 for all models. import torch import torch.nn as nn import timm def change_first_layer(m): for name, child in m.named_children(): if isinstance(child, nn.Conv2d): kwargs = { 'out_channels': child.out_channels, 'kernel_size': child.kernel_size, 'stride': child.stride, 'padding': child.padding, 'bias': False if child.bias == None else True } m._modules[name] = nn.Conv2d(15, **kwargs) return True else: if(change_first_layer(child)): return True return False x = torch.randn((8, 15, 224, 224)) m = timm.create_model('resnet200d', pretrained=False, num_classes=9) change_first_layer(m) print(m) print(m(x).shape)
Question: I already know How to import Julia packages into Python. However, now I have created my own simple Julia package with the following command: using Pkg;Pkg.generate("MyPack");Pkg.activate("MyPack");Pkg.add("StatsBase") where the file MyPack/src/MyPack.jl has the following contents: module MyPack using StatsBase function f1(x, y) return 3x + y end g(x) = StatsBase.std(x) export f1 end  Now I would like to load this Julia package in Python via juliacall and call f1 and g functions. I have already run pip3 install juliacall from command line. How do I call the above functions from Python?

Answer: You need to run the following code to load the MyPack package from Python via juliacall from juliacall import Main as jl from juliacall import Pkg as jlPkg jlPkg.activate("MyPack") # relative path to the folder where `MyPack/Project.toml` should be used here jl.seval("using MyPack")  Now you can use the function (note that calls to non exported functions require package name): >>> jl.f1(4,7) 19 >>> jl.f1([4,5,6],[7,8,9]).to_numpy() array([19, 23, 27], dtype=object) >>> jl.MyPack.g(numpy.arange(0,3)) 1.0  Note another option for calling Julia from Python that seems to be more difficult to configure as of today is the pip install julia package which is described here: I have a high-performant function written in Julia, how can I use it from Python?
Question: I have a CSV file in which I am using Python to parse. I found that some rows in the file have different number of columns. 001;Snow,Jon;19801201 002;Crom,Jake;19920103 003; ;Wise,Frank;19880303 <-- Invalid row 004;Wiseau,Tommy;4324;1323;2323 <-- Invalid row  I would like to write these invalid rows into a separate text file. I used this line of code to read from the file. df = pd.read_csv('names.csv', header=None,sep=';')  One solution I found here was to skip the problematic rows using the following code: data = pd.read_csv('file1.csv', on_bad_lines='skip')  I could change from 'skip' to 'warn', which will give the row number of the problematic row and skip the row. But this will return warning messages and not the row itself.

Answer: Since pandas 1.4.0 allows callable for on_bad_lines parameter - that allows you to apply a more sophisticated handling of bad lines.  New in version 1.4.0: callable, function with signature (bad_line: list[str]) -> list[str] | None that will process a single bad line. bad_line is a  list of strings split by the sep. If the function returns None, the bad line will be ignored. If the function returns a new list of strings with more elements than expected, a ParserWarning will be emitted while dropping extra elements. Only supported when engine="python"  So you can pass a custom function which will write an encountered bad line into specific file and return None (to skip that line on dataframe generation). from functools import partial def write_bad_line(line, fp, sep=','): fp.write(sep.join(line) + '\n') return None # return None to skip the line while processing bad_lines_fp = open('bad_lines.csv', 'a') df = pd.read_csv('test.csv', header=None, sep=';', engine='python', on_bad_lines=partial(write_bad_line, sep=';', fp=bad_lines_fp)) bad_lines_fp.close() print(df)  The output of the dataframe:  0 1 2 0 1 Snow,Jon 19801201 1 2 Crom,Jake 19920103   The contents of bad_lines.csv (via cat command): $ cat bad_lines.csv 003; ;Wise,Frank;19880303 004;Wiseau,Tommy;4324;1323;2323
Question: I have a data (user_data) that represent the number of examples in each class (here we have 5 classes), for example in first row, 16 represent 16 samples in class 1 for user1, 15 represent that there is 15 samples belong to class 2 for user 1, ect. user_data = np.array([ [16, 15, 14, 10, 0], [0, 13, 6, 15, 21], [12, 29, 1, 12, 1], [0, 0, 0, 0, 55]])  I used the following method to smooth all these frequencies to avoid issues with extreme values (0 or 1) by using Laplace smoothing where k=2.  Output: array([[0.29824561, 0.28070175, 0.26315789, 0.19298246, 0.01754386], [0.01754386, 0.24561404, 0.12280702, 0.28070175, 0.38596491], [0.22807018, 0.52631579, 0.03508772, 0.22807018, 0.03508772], [0.01754386, 0.01754386, 0.01754386, 0.01754386, 0.98245614]])  But I want to smooth only extreme values (0 or 1) in this data

Answer: I have noticed that your smoothing approach will cause P > 1, you are to clip or normalize the values later on: probs = user_data/55 alpha = (user_data+1)/(55+2) extreme_values_mask = (probs == 0) | (probs == 1) probs[extreme_values_mask] = alpha[extreme_values_mask]  Result: array([[0.29090909, 0.27272727, 0.25454545, 0.18181818, 0.01754386], [0.01754386, 0.23636364, 0.10909091, 0.27272727, 0.38181818], [0.21818182, 0.52727273, 0.01818182, 0.21818182, 0.01818182], [0.01754386, 0.01754386, 0.01754386, 0.01754386, 0.98245614]])  Extension: # Scale by sum. probs /= probs.sum(1).reshape((-1, 1)) print(probs) print(probs.sum(1)) [[0.28589342 0.26802508 0.25015674 0.17868339 0.01724138] [0.01724138 0.2322884 0.10721003 0.26802508 0.37523511] [0.21818182 0.52727273 0.01818182 0.21818182 0.01818182] [0.01666667 0.01666667 0.01666667 0.01666667 0.93333333]] [1. 1. 1. 1.]
Question: The order of operations in my code seems off...  numbers=[7, 6, 4] result = 1 for num in numbers: result *= num - 3 print(result)  In this code, I would expect the following to occur... result=1 result = 1 * 7 - 3 = 7 - 3 = 4 result = 4 * 6 - 3 = 24 - 3 = 21 result = 21 * 4 - 3 = 84 - 3 = 81  HOWEVER, running the program outputs result = 1 result = 1 * 7 - 3 = 1 * 4 = 4 result = 4 * 6 - 3 = 4 * 3 = 12 result = 12 * 4 - 3 = 12 * 1 = 12 Why with the *= operator is the order of operations altered? My understanding is it does not have any special properties it merely saves space instead of writing: result = result * num - 3 we get result *= num - 3 and for some reason.. (result = result * num - 3) != (result *= num - 3) 

Answer:  Why with the *= operator is the order of operations altered?  The *= is not an operator, it's delimiter. Check the '2. Lexical analysis' of 'The Python Language Reference':  The following tokens are operators: + - * ** / // % @ << >> & | ^ ~ := < > <= >= == !=    The following tokens serve as delimiters in the grammar: ( ) [ ] { } , : . ; @ = -> += -= *= /= //= %= @= &= |= ^= >>= <<= **=  The period can also occur in floating-point and imaginary literals. A sequence of three periods has a special meaning as an ellipsis literal. The second half of the list, the augmented assignment operators, serve lexically as delimiters, but also perform an operation.  Moreover, in your code you have the, augmented assignment statement, as per '7.2.1. Augmented assignment statements':  Augmented assignment is the combination, in a single statement, of a binary operation and an assignment statement: [...] An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target. The target is only evaluated once. An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. Unlike normal assignments, augmented assignments evaluate the left-hand side before evaluating the right-hand side. For example, a[i] += f(x) first looks-up a[i], then it evaluates f(x) and performs the addition, and lastly, it writes the result back to a[i].  The '6.16. Evaluation order' says:  Python evaluates expressions from left to right. Notice that while evaluating an assignment, the right-hand side is evaluated before the left-hand side.  You can also check the '6.17. Operator precedence' chapter, but it's not about the statements.
Question: Here is an async function generator by iterating a for loop. I expected this closure to respect the names from the outer scope. import asyncio coroutines = [] for param in (1, 3, 5, 7, 9): async def coro(): print(param ** 2) # await some_other_function() coroutines.append(coro) # The below code is not async, I made a poor mistake. # Michael Szczesny's answer fixes this. for coro in coroutines: asyncio.run(coro())  While I was expecting the results to be 1, 9, 25, 49, 81 after running, here is the actual output: 81 81 81 81 81  Unexpectedly here, the name param has taken the same value each time. Can you explain why this happens and how I can achieve the task of creating lots of async functions in a for loop with names binding correctly?

Answer: The included code does not run asynchronously or uses coro as a closure. Functions are evaluated at runtime in python. An asynchronous solution would look like this import asyncio def create_task(param): async def coro(): await asyncio.sleep(1) # make async execution noticeable print(param ** 2) return coro # return closure capturing param async def main(): await asyncio.gather(*[create_task(param)() for param in (1,3,5,7,9)]) asyncio.run(main())  Output after 1 second 1 9 25 49 81
Question: I need a parametric form for a matplotlib.path.Path. So I used the .vertices attribute, and it works fine except that the number of points given is too low for the use I want. Here is a code to illustrate : import numpy as np from matplotlib import pyplot as plt import matplotlib.patches as mpat fig, ax = plt.subplots() ax.set(xlim=(-6, 6), ylim=(-6, 6)) # generate a circular path circle = mpat.Arc((0, 0), 10, 10, theta1=20, theta2=220, color='green') path = circle.get_transform().transform_path(circle.get_path()).cleaned().vertices[:-3] # get_path is not enough because of the transformation, so we apply to the path the same transformation as the circle has got from the identity circle ax.add_patch(circle) # plot path vertices plt.scatter(x=path[:, 0], y=path[:, 1], color='red', s=2) shape = len(path) plt.show()  How to increase the number of points (red) to fetch better the path (green)? Or, how can I increase the len of path? Thanks in advance!

Answer: It's not ideal, but you can just loop over many shorter arcs, e.g.,: import numpy as np from matplotlib import pyplot as plt import matplotlib.patches as mpat fig, ax = plt.subplots() ax.set(xlim=(-6, 6), ylim=(-6, 6)) # generate a circular path dr = 30 # change this to control the number of points path = np.empty((0, 2)) for i in range(20, 221, dr): circle = mpat.Arc((0, 0), 10, 10, theta1=i, theta2=(i + dr), color='green') tmppath = circle.get_transform().transform_path(circle.get_path()).cleaned().vertices[:-1] path = np.vstack((path, tmppath[:, 0:2])) ax.add_patch(circle) # plot path vertices plt.scatter(x=path[:, 0], y=path[:, 1], color='red', s=2) plt.show()  Update Another option is to hack the Arc class, so that during initialisation, the Path.arc takes in the n keyword. E.g., from matplotlib.patches import Path, Arc class NewArc(Arc): def __init__(self, xy, width, height, angle=0.0, theta1=0.0, theta2=360.0, n=30, **kwargs): """ An Arc class with the n keyword argument """ fill = kwargs.setdefault('fill', False) if fill: raise ValueError("Arc objects can not be filled") super().__init__(xy, width, height, angle=angle, **kwargs) self.theta1 = theta1 self.theta2 = theta2 (self._theta1, self._theta2, self._stretched_width, self._stretched_height) = self._theta_stretch() # add in the n keyword here self._path = Path.arc(self._theta1, self._theta2, n=n)  The original code (without the for loop) can then be used and the number of points increased, e.g., from matplotlib import pyplot as plt fig, ax = plt.subplots() ax.set(xlim=(-6, 6), ylim=(-6, 6)) # generate a circular path using NewArc and the n keyword circle = NewArc((0, 0), 10, 10, theta1=20, theta2=220, n=40, color='green') path = circle.get_transform().transform_path(circle.get_path()).cleaned().vertices[:-3] ax.add_patch(circle) # plot path vertices ax.scatter(x=path[:, 0], y=path[:, 1], color='red', s=2) fig.show()
Question: I have a pandas dataframe and I want to add to it new features, like this: Say I have features X_1,X_2,X_3 and X_4, then I want to add X_1 * X_2, X_1 * X_3, X_1 * X_4, and similarly X_2 * X_3, X_2 * X_4 and X_3 * X_4. I want to add them, not replace the original features. How do I do that?

Answer: for c1, c2 in combinations(df.columns, r=2): df[f"{c1} * {c2}"] = df[c1] * df[c2]  you can take every r = 2 combination of the columns, multiply them and assign. Example run: In [66]: df Out[66]: x1 y1 x2 y2 0 20 5 22 10 1 25 8 27 2 In [67]: from itertools import combinations In [68]: for c1, c2 in combinations(df.columns, r=2): ...: df[f"{c1} * {c2}"] = df[c1] * df[c2] ...: In [69]: df Out[69]: x1 y1 x2 y2 x1 * y1 x1 * x2 x1 * y2 y1 * x2 y1 * y2 x2 * y2 0 20 5 22 10 100 440 200 110 50 220 1 25 8 27 2 200 675 50 216 16 54   Another way via sklearn.preprocessing.PolynomialFeatures: In [74]: df Out[74]: x1 y1 x2 y2 0 20 5 22 10 1 25 8 27 2 In [75]: from sklearn.preprocessing import PolynomialFeatures In [76]: poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False) In [77]: poly.fit_transform(df) Out[77]: array([[ 20., 5., 22., 10., 100., 440., 200., 110., 50., 220.], [ 25., 8., 27., 2., 200., 675., 50., 216., 16., 54.]]) In [78]: new_columns = df.columns.tolist() + [*map(" * ".join, combinations(df.columns, r=2))] In [79]: df = pd.DataFrame(poly.fit_transform(df), columns=new_columns) In [80]: df Out[80]: x1 y1 x2 y2 x1 * y1 x1 * x2 x1 * y2 y1 * x2 y1 * y2 x2 * y2 0 20.0 5.0 22.0 10.0 100.0 440.0 200.0 110.0 50.0 220.0 1 25.0 8.0 27.0 2.0 200.0 675.0 50.0 216.0 16.0 54.0
Question: I have created many interactive graphs on Plotly Dash (using Python) that are working perfectly on the local port. But I would like to integrate this into other products on a website. Is this possible? What would be the best way to display exactly what I'm seeing on my local port? I have read that we may use <iframe> for this, but I don't have expertise to say if this is the best solution

Answer: This is a somewhat open-ended question, but I would recommend following @Federico Tartarini's excellent YouTube guide here that will allow you to deploy your plotly-dash app to Google Cloud. His video will provide more exact instructions than I'll be able to type in this answer, but I'll summarize the main points: You’ll put your app.py file inside of a dash project with this structure: Dash project/ ├─ venv/ │ ├─ Dockerfile/ │ ├─ app.py/ │ ├─ requirements.txt/ ├─ assets/ ├─ data/  Then you'll run commands to deploy your project (including the dash app) to Google Cloud – and Google Cloud will follow instructions inside the Dockerfile and requirements.txt to create a virtual environment that can run your app.py file. Once the app is deployed to the cloud, you'll be able to access the app and embed it on your webpage. This is how I deployed my trigonometric interpolation plotly-dash app to Google Cloud and embedded it on the projects section of my personal webpage.
Question: The package PyAudioWPatch is shown as available on PyPi with a big old green check mark. https://pypi.org/project/PyAudioWPatch/ However when I try to install it, I am getting the following error: % pip install PyAudioWPatch ERROR: Could not find a version that satisfies the requirement PyAudioWPatch (from versions: none) ERROR: No matching distribution found for PyAudioWPatch  For context: % python -V; pip -V Python 3.9.13 pip 22.3.1 from /Users/petertoth/Documents/Desktop_record_sum/py/venv/lib/python3.9/site-packages/pip (python 3.9)  Why is this the case?

Answer: The project has only wheels for Windows, and your system is not Windows, hence the error. And I should assume that to be by design, because it declares to be a PortAudio fork with WASAPI loopback support. As WASAPI is a Windows thing, it does not make sense to install it on a non Windows system. IMHO, you'd better install the original project. If you really want this one, then you should try to get it from GitHub
Question: I have the following formula that I would like to evaluate: import math import numpy as np formula = 'np.e**x + math.erf(x) + np.pi + math.erf(u)'  I can easily then evaluate the formula for given float values of x and u and eval() recognizes math.erf, np.pi and np.e. For example: x=1.0; u=0.3; eval(formula)  yields 7.03. But, I want x and u to be arrays. I tried to use eval with a dictionary following this post.: var = {'x':np.array([1,1]),'u':np.array([0.1,0.2])} eval(formula, var)  which yields error messages, 'np' and 'math' are not defined, which was not the case above when eval was used without a dict. The same error messages are also obtained when 'x' and 'u' are set to floats instead of array with the var dictionary. Also, there are no problems if the dictionary is used with a 'formula' without np. and math., e.g. " formula = 'x + u'.  Does anybody have an idea how can I evaluate formula containing np.e, math.erf, etc., when x and u are arrays? P.S. I am using Python 3.8.

Answer: Passing var as globals to eval() means the global names np and math are not available to the expression (formula). Simply pass var as locals instead. var = {'x': 1.0, 'u': 0.3} eval(formula, None, var) # -> 7.031202034457681  Note that this code doesn't work with arrays since the math module only works on scalars.
Question: I want to apply MinmaxScaler on a number of pandas DataFrame 'together'. Meaning that I want the scaler to perform on all data in those columns, not separately on each column. My DataFrame has 20 columns. I want to apply the scaler on 12 of the columns at the same time. I have already read this. But it does not solve my problem since it acts on each column separately.

Answer: IIUC, you want the sklearn scaler to fit and transform multiple columns with the same criteria (in this case min and max definitions). Here is one way you can do this -  You can save the initial shape of the columns and then transform the numpy array of those columns into a 1D array from a 2D array. Next you can fit your scaler and transform this 1D array Finally you can use the old shape to reshape the array back into the n columns you need and save them  The advantage of this approach is that this works with any of the sklearn scalers you need to use, MinMaxScaler, StandardScaler etc. import pandas as pd from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler() dfTest = pd.DataFrame({'A':[14.00,90.20,90.95,96.27,91.21], 'B':[103.02,107.26,110.35,114.23,114.68], 'C':['big','small','big','small','small']}) cols = ['A','B'] old_shape = dfTest[cols].shape #(5,2) dfTest[cols] = scaler.fit_transform(dfTest[cols].to_numpy().reshape(-1,1)).reshape(old_shape) print(dfTest)   A B C 0 0.000000 0.884188 big 1 0.756853 0.926301 small 2 0.764303 0.956992 big 3 0.817143 0.995530 small 4 0.766885 1.000000 small
Question: I am setting the options like this pd.options.display.max_columns = None  When I try to print the DataFrame, I get truncated columns: <class 'pandas.core.frame.DataFrame'> Index(['contractSymbol', 'strike', 'currency', 'lastPrice', 'change', 'volume', 'bid', 'ask', 'contractSize', 'lastTradeDate', 'impliedVolatility', 'inTheMoney', 'openInterest', 'percentChange'], dtype='object') contractSymbol strike currency \ symbol expiration optionType TSLA 2022-12-30 calls TSLA221230C00050000 50.00 USD calls TSLA221230C00065000 65.00 USD  How do I show all columns in one row?

Answer: I think you need to set the display size to a larger value. According to the documentation, display.max_columns defines the behavior taken when max_cols is exceeded. I'm not sure if this is referring to the number of columns, or the width of all columns. In either case, setting display.width to a larger value: pd.options.display.width = 120  would probably fix your issue. The default is 80 characters, which is about what you have there before the value is written to a newline. If your editor is in a terminal window that you can resize, you could also try doing that.