eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was wondering if anyone knew how the build() function works from the tf.keras.layers.Layer class under the hood. According to the documentation:  build is called when you know the shapes of the input tensors and can do the rest of the initialization  so to me it seems like the class is behaving similar to this: class MyDenseLayer: def __init__(self, num_outputs): self.num_outputs = num_outputs def build(self, input_shape): self.kernel = self.add_weight("kernel", shape=[int(input_shape[-1]), self.num_outputs]) def __call__(self, input): self.build(input.shape) ## build is called here when input shape is known return tf.matmul(input, self.kernel)  I can't imagine build() would be called for ever __call__, but it is the only place where the input is passed in. Does anyone know how exactly this works under the hood?

Answer: The Layer.build() method is typically used to instantiate the weights of the layer. See the source code for tf.keras.layers.Dense for an example, and note that the weight and bias tensors are created in that function. The Layer.build() method takes an input_shape argument, and the shape of the weights and biases often depend on the shape of the input. The Layer.call() method, on the other hand, implements the forward-pass of the layer. You do not want to overwrite __call__, because that is implemented in the base class tf.keras.layers.Layer. In a custom layer, you should implement call(). Layer.call() does not call Layer.build(). However, Layer().__call__() does call it if the layer has not been built yet (source), and that will set an attribute self.built = True to prevent Layer.build() from being called again. In other words, Layer.__call__() only calls Layer.build() the first time it is called.
Question: I'm running into an issue while calculating the loss for my Neural Net. I'm not sure why the program expects a long object because all my Tensors are in float form. I looked at threads with similar errors and the solution was to cast Tensors as floats instead of longs, but that wouldn't work in my case because all my data is already in float form when passed to the network. Here's my code: # Dataloader from torch.utils.data import Dataset, DataLoader class LoadInfo(Dataset): def __init__(self, prediction, indicator): self.prediction = prediction self.indicator = indicator def __len__(self): return len(self.prediction) def __getitem__(self, idx): data = torch.tensor(self.indicator.iloc[idx, :],dtype=torch.float) data = torch.unsqueeze(data, 0) label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.float) sample = {'data': data, 'label': label} return sample # Trainloader test_train = LoadInfo(train_label, train_indicators) trainloader = DataLoader(test_train, batch_size=64,shuffle=True, num_workers=1,pin_memory=True) # The Network class NetDense2(nn.Module): def __init__(self): super(NetDense2, self).__init__() self.rnn1 = nn.RNN(11, 100, 3) self.rnn2 = nn.RNN(100, 500, 3) self.fc1 = nn.Linear(500, 100) self.fc2 = nn.Linear(100, 20) self.fc3 = nn.Linear(20, 3) def forward(self, x): x1, h1 = self.rnn1(x) x2, h2 = self.rnn2(x1) x = F.relu(self.fc1(x2)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x # Allocate / Transfer to GPU dense2 = NetDense2() dense2.cuda() # Optimizer import torch.optim as optim criterion = nn.CrossEntropyLoss() # specify the loss function optimizer = optim.SGD(dense2.parameters(), lr=0.001, momentum=0.9,weight_decay=0.001) # Training dense2.train() loss_memory = [] for epoch in range(50): # loop over the dataset multiple times running_loss = 0.0 for i, samp in enumerate(trainloader): # get the inputs ins = samp['data'] targets = samp['label'] tmp = [] tmp = torch.squeeze(targets.float()) ins, targets = ins.cuda(), tmp.cuda() # zero the parameter gradients optimizer.zero_grad() # forward + backward + optimize outputs = dense2(ins) loss = criterion(outputs, targets) # The loss loss.backward() optimizer.step() # keep track of loss running_loss += loss.data.item()  I get the error from above in the line " loss = criterion(outputs, targets) "

Answer: As per the documentation and official example at pytorch webpage, The targets passed to nn.CrossEntropyLoss() should be in torch.long format # official example import torch import torch.nn as nn loss = nn.CrossEntropyLoss() input = torch.randn(3, 5, requires_grad=True) target = torch.empty(3, dtype=torch.long).random_(5) # if you will replace the dtype=torch.float, you will get error output = loss(input, target) output.backward()  update this line in your code as label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.long) #updated torch.float to torch.long
Question: First of all, I'm new to Django, so please be nice with me :D I'm currently adapting .py files for Django 3 because the files I have are compatible for Django 2. So, some changes have been made for the new version and in a file, it's written : @wraps(view_func, assigned=available_attrs(view_func))  With the import : from django.utils.decorators import available_attrs  I searched for an adaptation of available_attrs, and I quickly found that it has been removed for the new version. And when I launch the code, I have this : ImportError : cannot import name 'available_attrs' from 'django.utils.decorators'  So I was wondering what should I write instead of available_attrs to make it work ? PS : Sorry for my bad english

Answer: available_attrs() only ever existed to help bridge between Python 2 and Python 3. This is documented in the Django 3.0 release notes:  Removed private Python 2 compatibility APIs While Python 2 support was removed in Django 2.0, some private APIs weren’t removed from Django so that third party apps could continue using them until the Python 2 end-of-life. Since we expect apps to drop Python 2 compatibility when adding support for Django 3.0, we’re removing these APIs at this time.  [...] django.utils.decorators.available_attrs() - This function returns functools.WRAPPER_ASSIGNMENTS   If the @wraps() in your sample line is the standard functools.wraps() decorator, then you can just entirely remove assigned=available_attrs(...), because functools.WRAPPER_ASSIGNMENTS is the default value for assigned: @wraps(view_func)  otherwise, just use functools.WRAPPER_ASSIGNMENTS directly.
Question: I need to open a gzipped file, that has a parquet file inside with some data. I am having so much trouble trying to print/read what is inside the file. I tried the following: with gzip.open("myFile.parquet.gzip", "rb") as f: data = f.read()  This does not seem to work, as I get an error that my file id not a gz file. Thanks!

Answer: You can use read_parquet function from pandas module:  Install pandas and pyarrow:  pip install pandas pyarrow   use read_parquet which returns DataFrame:  data = read_parquet("myFile.parquet.gzip") print(data.count()) # example of operation on the returned DataFrame
Question: I have a distplot and I would like to plot a mean line that goes from 0 to the y value of the mean frequency. I want to do this, but have the line stop at when the distplot does. Why isn't there a simple parameter that does this? It would be very useful. I have some code that gets me almost there: plt.plot([x.mean(),x.mean()], [0, *what here?*])  This code plots a line just as I'd like except for my desired y-value. What would the correct math be to get the y max to stop at the frequency of the mean in the distplot? An example of one of my distplots is below using 0.6 as the y-max. It would be awesome if there was some math to make it stop at the y-value of the mean. I have tried dividing the mean by the count etc. 

Answer: Update for the latest versions of matplotlib (3.3.4) and seaborn (0.13.3): the kdeplot with shade=True now doesn't create a line object anymore. To get the same outcome as before, setting fill=False will still create the line object. The curve can then be filled with ax.fill_between(). The code below is changed accordingly. (Use the revision history to see the older versions.) ax.lines[0] gets the curve of the kde, of which you can extract the x and y data. np.interp then can find the height of the curve for a given x-value: import numpy as np import matplotlib.pyplot as plt import seaborn as sns x = np.random.normal(np.tile(np.random.uniform(10, 30, 5), 50), 3) ax = sns.kdeplot(x, fill=False, color='crimson') kdeline = ax.lines[0] mean = x.mean() xs = kdeline.get_xdata() ys = kdeline.get_ydata() height = np.interp(mean, xs, ys) ax.vlines(mean, 0, height, color='crimson', ls=':') ax.fill_between(xs, 0, ys, facecolor='crimson', alpha=0.2) plt.show()   The same approach can be extended to show the mean together with the standard deviation, or the median and the quartiles: import matplotlib.pyplot as plt import seaborn as sns import numpy as np x = np.random.normal(np.tile(np.random.uniform(10, 30, 5), 50), 3) fig, axes = plt.subplots(ncols=2, figsize=(12, 4)) for ax in axes: sns.kdeplot(x, fill=False, color='crimson', ax=ax) kdeline = ax.lines[0] xs = kdeline.get_xdata() ys = kdeline.get_ydata() if ax == axes[0]: middle = x.mean() sdev = x.std() left = middle - sdev right = middle + sdev ax.set_title('Showing mean and sdev') else: left, middle, right = np.percentile(x, [25, 50, 75]) ax.set_title('Showing median and quartiles') ax.vlines(middle, 0, np.interp(middle, xs, ys), color='crimson', ls=':') ax.fill_between(xs, 0, ys, facecolor='crimson', alpha=0.2) ax.fill_between(xs, 0, ys, where=(left <= xs) & (xs <= right), interpolate=True, facecolor='crimson', alpha=0.2) # ax.set_ylim(ymin=0) plt.show()   PS: for the mode of the kde:  mode_idx = np.argmax(ys) ax.vlines(xs[mode_idx], 0, ys[mode_idx], color='lime', ls='--')
Question: The url() function in django has been deprecated since version 3.1. Here's how backwards compatibility is being handled; def url(regex, view, kwargs=None, name=None): warnings.warn( 'django.conf.urls.url() is deprecated in favor of ' 'django.urls.re_path().', RemovedInDjango40Warning, stacklevel=2, ) return re_path(regex, view, kwargs, name)  For now, re_path() is returned when the url() function is called. When the function is completely removed, will the projects that use it have to change their source code?

Answer:  will the projects that use it have to change their source code?  Yes, if they upgrade to django-4.0, url will no longer be available. Typically if something is marked deprecated, it is removed two versions later, so in django-4.0, since after django-3.2, django-4.0 will be released. If you thus have an active project, you eventually will upgrade to Django-4.0 or further, and thus should make use of re_path(…) [Django-doc] instead. The idea is thus to give users time to adapt the code accordingly, and keep the application running. But one should eventually fix the deprecation warnings, since after ~16 months, it is removed in the newest Django versions.
Question: I have a pyarrow table name final_table of shape 6132,7 I want to add column to this table  list_ = ['IT'] * 6132 final_table.append_column('COUNTRY_ID', list_)  but I am getting following error ArrowInvalid: Added column's length must match table's length. Expected length 6132 but got length 12264

Answer: According to the documentation: Append column at end of columns. Parameters field (str or Field) – If a string is passed then the type is deduced from the column data. column (Array, list of Array, or values coercible to arrays) – Column data. Returns pyarrow.Table – New table with the passed column added.  I think pyarrow is assuming that you're providing a list of Array. To avoid the confusion you should pass an arrow array instead col_a = pa.array([1, 2, 3], pa.int32()) col_b = pa.array(["X", "Y", "Z"], pa.string()) table = pa.Table.from_arrays( [col_a, col_b], schema=pa.schema([ pa.field('a', col_a.type), pa.field('b', col_b.type), ]) ) table = table.append_column('COUNTRY_ID', pa.array(['IT'] * len(table), pa.string()))
Question: I’m slightly disappointed that np.inf // 2 evaluates to np.nan and not to np.inf, as is the case for normal division. Is there a reason I’m missing why nan is a better choice than inf?

Answer: I'm going to be the person who just points at the C level implementation without any attempt to explain intent or justification: *mod = fmod(vx, wx); div = (vx - *mod) / wx;  It looks like in order to calculate divmod for floats (which is called when you just do floor division) it first calculates the modulus and float('inf') %2 only makes sense to be NaN, so when it calculates vx - mod it ends up with NaN so everything propagates nan the rest of the way. So in short, since the implementation of floor division uses modulus in the calculation and that is NaN, the result for floor division also ends up NaN
Question: Documentation for Django 3.1 says this about async views:  The main benefits are the ability to service hundreds of connections without using Python threads. This allows you to use slow streaming, long-polling, and other exciting response types.  I believe that "slow streaming" means we could implement an SSE view without monopolizing a thread per client, so I tried to sketch a simple view, like so: async def stream(request): async def event_stream(): while True: yield 'data: The server time is: %s\n\n' % datetime.datetime.now() await asyncio.sleep(1) return StreamingHttpResponse(event_stream(), content_type='text/event-stream')  (note: I adapted the code from this response) Unfortunately, when this view is invoked, it raises the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/asgiref/sync.py", line 330, in thread_handler raise exc_info[1] File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 38, in inner response = await get_response(request) File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 231, in _get_response_async response = await wrapped_callback(request, *callback_args, **callback_kwargs) File "./chat/views.py", line 144, in watch return StreamingHttpResponse(event_stream(), content_type='text/event-stream') File "/usr/local/lib/python3.7/site-packages/django/http/response.py", line 367, in __init__ self.streaming_content = streaming_content File "/usr/local/lib/python3.7/site-packages/django/http/response.py", line 382, in streaming_content self._set_streaming_content(value) File "/usr/local/lib/python3.7/site-packages/django/http/response.py", line 386, in _set_streaming_content self._iterator = iter(value) TypeError: 'async_generator' object is not iterable  To me, this shows that StreamingHttpResponse doesn't currently support async generators. I tried to modify StreamingHttpResponse to use async for but I wasn't able to do much. Any idea how I could do that?

Answer: This is an old question but it came up on a Google result since I was looking for a solution to the same issue. In the end I found this repo https://github.com/valberg/django-sse - which uses async views in Django 4.2 to stream via SSE (specifically see here). I understand this is a recent addition to Django so I hope it helps anyone else looking for an answer.
Question: Any documentation I've found about this topic mentions that the "only" requirement to build a deb package is to have a correct setup.py (and requirements.txt). For instance in dh-virtualenv tutorial, stdeb documentation and the Debian's library style guide for python. But nowadays new (amazing) tools like poetry allow to develop (and upload to PyPI) python projects without any setup.py (this file and several others including requirements.txt are all replaced by pyproject.toml). I believe flit allows this too. I have developed a python project managed by poetry and would like to package it for Ubuntu/Debian. I guess, as a workaround I can still write a setup.py file that would take its values from pyproject.toml and a requirements.txt file (written by hand using values from poetry.lock). But, is there a way to do this without any setup.py file?

Answer: setuptools, and the setup.py file that it requires, has been the de-facto packaging standard in python for the longest time. The new package managers you mention were enabled by the introduction of PEP 517 and PEP 518 (or read this for a high-level description on the topic), which provide a standardized way of specifying the build backend without the need of a setup.py (and the ensuing hen-egg problem where you already need setuptools to correctly parse it, but might also need to specify which version of setuptools you want for the build). Anyway, it's all still very fresh, and the linux packaging community hasn't fully caught up yet. I found this discussion on the debian bug tracker, and the rpm side sums it up neatly over here. The short answer is to just wait a while until the toolchain catches up to the new standards, and google debian packaging pep517 support every now and then.  As a workaround for poetry-build packages specifically, you can use dephell to generate the setup.py, and poetry to generate the requirements.txt for you to keep using legacy tools: dephell deps convert --from=poetry --to=setuppy poetry export -f requirements.txt -o requirements.txt  And, during the build, tell your pyproject.tom that you plan to use setuptools for the build instead of poetry: [build-system] requires = ["setuptools >= 40.6.0", "wheel"] build-backend = "setuptools.build_meta"
Question: I'm attempting to import a script from my Items file but I keeps on getting an error from .Items.Quest1_items import *  gives from .Items.Quest1_items import * # ImportError: attempted relative import with no known parent package # Process finished with exit code 1  Here my project tree, I'm running the script from the main.py file Quest1/ | |- main.py | |- Items/ | |- __init__.py | |- Quest1_items.py 

Answer: Remove the dot from the beginning. Relative paths with respect to main.py are found automatically. from Items.Quest1_items import *
Question: I have created the following Enum: from enum import Enum class Action(str, Enum): NEW_CUSTOMER = "new_customer" LOGIN = "login" BLOCK = "block"  I have inherited from str, too, so that I can do things such as: action = "new_customer" ... if action == Action.NEW_CUSTOMER: ...  I would now like to be able to check if a string is in this Enum, such as: if "new_customer" in Action: ....  I have tried adding the following method to the class: def __contains__(self, item): return item in [i for i in self]  However, when I run this code: print("new_customer" in [i for i in Action]) print("new_customer" in Action)  I get this exception: True Traceback (most recent call last): File "/Users/kevinobrien/Documents/Projects/crazywall/utils.py", line 24, in <module> print("new_customer" in Action) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/enum.py", line 310, in __contains__ raise TypeError( TypeError: unsupported operand type(s) for 'in': 'str' and 'EnumMeta' 

Answer: I just bumped into this problem today (2020-12-09); I had to change a number of subpackages for Python 3.8. Perhaps an alternative to the other solutions here is the following, inspired by the excellent answer here to a similar question, as well as @MadPhysicist's answer on this page: from enum import Enum, EnumMeta class MetaEnum(EnumMeta): def __contains__(cls, item): try: cls(item) except ValueError: return False return True class BaseEnum(Enum, metaclass=MetaEnum): pass class Stuff(BaseEnum): foo = 1 bar = 5  Tests (python >= 3.7; tested up to 3.10): >>> 1 in Stuff True >>> Stuff.foo in Stuff True >>> 2 in Stuff False >>> 2.3 in Stuff False >>> 'zero' in Stuff False
Question: Context I'm trying to validate/parse some data with pydantic. I want to specify that the dict can have a key daytime, or not. If it does, I want the value of daytime to include both sunrise and sunset. e.g. These should be allowed: { 'type': 'solar', 'daytime': { 'sunrise': 4, # 4am 'sunset': 18 # 6pm } }  And { 'type': 'wind' # daytime key is omitted }  And { 'type': 'wind', 'daytime': None }  But I want to fail validation for { 'type': 'solar', 'daytime': { 'sunrise': 4 } }  Because this has a daytime value, but no sunset value. MWE I've got some code that does this. If I run this script, it executes successfully. from pydantic import BaseModel, ValidationError from typing import List, Optional, Dict class DayTime(BaseModel): sunrise: int sunset: int class Plant(BaseModel): daytime: Optional[DayTime] = None type: str p = Plant.parse_obj({'type': 'wind'}) p = Plant.parse_obj({'type': 'wind', 'daytime': None}) p = Plant.parse_obj({ 'type': 'solar', 'daytime': { 'sunrise': 5, 'sunset': 18 }}) try: p = Plant.parse_obj({ 'type': 'solar', 'daytime': { 'sunrise': 5 }}) except ValidationError: pass else: raise AssertionError("Should have failed")  Question What I'm wondering is, is this how you're supposed to use pydantic for nested data? I have lots of layers of nesting, and this seems a bit verbose. Is there any way to do something more concise, like: class Plant(BaseModel): daytime: Optional[Dict[('sunrise', 'sunset'), int]] = None type: str 

Answer: Pydantic create_model function is what you need: from pydantic import BaseModel, create_model class Plant(BaseModel): daytime: Optional[create_model('DayTime', sunrise=(int, ...), sunset=(int, ...))] = None type: str
Question: I'm a beginner in Python. Recently, I finished the basics, and now I'm trying to make some GUI applications. I have found many cases where we use config() and configure(). But what is the difference between config() and configure()? I mean, in what cases should config() be used, and in what cases should configure() be used? I have a small portion of my code here. Code where Configure is used: fontStyle = tkFont.Font(family="comic sans ms", size=15) def zoomin_1(): fontsize=fontStyle['size'] fontStyle.configure(size=fontsize+5) def zoomout_1(): fontsize = fontStyle['size'] fontStyle.configure(size=fontsize - 5) def default_1(): fontStyle.configure(size=10)  Code where config is used: root.config(menu=mainmenu) mainmenu.add_cascade(label="Edit", menu=Edit_menu) Format_menu = Menu(mainmenu, tearoff = False) Format_menu.add_command(label="Word Wrap", command= test_fun) Format_menu.add_command(label="Font", command= test_fun) root.config(menu=mainmenu) mainmenu.add_cascade(label="Format", menu=Format_menu)  It would of great help if someone would clear this doubt.

Answer: Both are exactly the same, the only difference is, the difference in the name, I would just reccomend using .config() just to save a few typing characters ;-)
Question: I have a certain piece of code that looks like this: # file1.py from module import Object def method(): o = Object("param1") o.do_something("param2")  I have unittests that look like this: @patch("file1.Object") class TestFile(unittest.TestCase): def test_call(self, obj): ...  I can do obj.assert_called_with() in the unittest to verify that the constructor was called with certain parameters. Is it possible to verify that obj.do_something was called with certain parameters? My instinct is no, as the Mock is fully encapsulated within Object, but I was hoping there might be some other way.

Answer: You can do this, because the arguments are passed to the mock object. This should work: @patch("file1.Object") class TestFile: def test_call(self, obj): method() obj.assert_called_once_with("param1") obj.return_value.do_something.assert_called_once_with("param2")  obj.return_value is the Object instance (which is a MagickMock object with the Object spec), and do_something is another mock in that object that is called with the given parameter. As long as you are just passing arguments to mock objects, the mock will record this and you can check it. What you don't have is any side effects from the real function calls - so if the original do_something would call another function, this cannot be checked.