akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to use `torch.gather` with 3D index and 3D input

gc625: <p>I have an <code>idx</code> tensor with shape <code>torch.Size([2, 80000, 3])</code>, which corresponds to batch, points, and indices of 3 elements (from 0 to 512) from the <code>feat</code> tensor  with shape <code>torch.Size([2, 513, 2])</code>.</p>
<p>I cant seem to find a way to use <code>torch.gather</code> to index <code>feat</code> with <code>idx</code> with a resulting tensor with shape <code>[2,8000,3,2]</code>.</p>

ptrblck: <p>Unfortunately you haven’t shared a reference code (even if slow) which I could use to compare my approach against, but this might work:</p>
<pre><code class="lang-python">idx = torch.randint(0, 513, [2, 80000, 3])
x = torch.randn(2, 513, 2)

out = x[torch.arange(x.size(0))[:, None, None], idx]
print(out.shape)
# torch.Size([2, 80000, 3, 2])
</code></pre>

gc625: <p>ah apologies. After awhile I did manage to find a way to do it (I think), although a bit more clunkly than your implementation:</p>
<pre><code class="lang-auto">
idx = torch.randint(0, 513, [2, 80000, 3])
feats = torch.randn(2, 513, 2)

B,N,K = idx.shape
temp_idx = idx.reshape(B,N*K,1)
gather_idx = torch.concat((temp_idx,temp_idx),dim=2)
gathered_features = torch.gather(feats,1,gather_idx)
final_feats = gathered_features.reshape((B,80000,3,2))
</code></pre>
<p>any ideas about which approach is better?</p>

ptrblck: <p>Both methods should yield the same output, but you might want to profile both approaches.<br>
On the CPU I’m getting a faster result using my direct indexing:</p>
<pre><code class="lang-python">def fun1(x, idx):
    out = x[torch.arange(x.size(0))[:, None, None], idx]
    return out

def fun2(x, idx):
    B,N,K = idx.shape
    temp_idx = idx.reshape(B,N*K,1)
    gather_idx = torch.concat((temp_idx,temp_idx),dim=2)
    gathered_features = torch.gather(x,1,gather_idx)
    gathered_features = gathered_features.view(2, 80000, 3, 2)
    return gathered_features 
    
    
idx = torch.randint(0, 513, [2, 80000, 3])
x = torch.randn(2, 513, 2)

out1 = fun1(x, idx)
out2 = fun2(x, idx)

print((out1 - out2).abs().max())
# tensor(0.)

%timeit fun1(x, idx)
# 355 µs ± 97.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit fun2(x, idx)
# 1.17 ms ± 112 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>
<p>Note that you would need to add synchronizations in case you want to profile the code on the GPU.</p>
NCCL Single machine multi card training error

CodeCat: <p>error info：To avoida data incosistency, we are taking the entire process down<br>
In the training process, some training samples will report errors in the calculation process due to the use of some random cutting functions. Therefore, I choose to skip these training samples. The consequence of this operation is that errors will be reported during single machine multi card training, and there will be no problems during single machine single card training. Just like the error message, it should be the data inconsistency caused by skipping the data sample. How to solve this problem? Is there any way to make all video cards skip a training together?</p>

CodeCat: <p>I’m very worried. Could you please answer this question for me</p>
Manually divide data to different GPUs in DataParallel

jasperhyp: <p>I have several GPUs on the server but they are shared with others, so some might be more available than others.  How can I let data go to different GPUs in an imbalanced way?</p>

huahuanZ: <p>If you would like to use the GPUs exclusively (one card won’t be shared by two persons), configuring environment variable <code>CUDA_VISIBLE_DEVICES=0,1</code> to specify using GPU 0 and 1.</p>
<p>If what you mean is to dive a mini-batch unevenly across GPUs, things are more complicated:<br>
For DP, I’m not sure if there is an ‘elegant’ way. A possible hack way is to manually split the mini-batch before calling the forward() function and wrap it in a list like <code>[batch[:10], batch[10:50]]</code>. Inside the forward() function, each GPU fetches corresponding sub-batch according to its gpu id.</p>

jasperhyp: <p>Yes exactly! Am thinking about the latter. Do you mean when I send <code>x.to(device)</code> in each batch, I should explicitly do <code>x[:10].to(torch.device('cuda:1'))</code> and <code>x[10:].to(torch.device('cuda:0'))</code> instead? I guess not in <code>forward()</code> ?</p>

huahuanZ: <p>I wrote an example</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.param = nn.parameter.Parameter(torch.empty(1))

    def forward(self, x):
        x = torch.tensor(
            x[self.param.device.index],
            device=self.param.device.index
        )
        print(x)


if __name__ == "__main__":
    model = MyModel().cuda()
    model = nn.DataParallel(model)

    x = np.arange(32)
    x = [x[:10], x[10:]]
    model(x)

</code></pre>
<p>You should pass non-tensor type to forward(), otherwise the DP would automatically split it. Here I passed a np.array to forward. Splitting is conducted before calling the forward() function.</p>
<p>And note, if the batch is split unevenly, you can’t get loss scalar by <code>loss_list.mean()</code>, instead using <code>loss = (loss_list * n_sub_batch).sum() / n_tot_batch</code></p>
Tensor.conj() for python tensor and then operation in c++ api

tsp_t: <p>Tensor.conj() applied in python3 was not applied when operation in c++. I would like to know how to properly use this in C++ operations.</p>
<p>I want to implement it as general as possible, so I want to solve it without the user using conj_physical instead of conj or additionally calling resolve_conj in the code.</p>
<p>part of the code:</p>
<p>conj_physical<br>
python3:</p>
<pre><code class="lang-auto">A = (...).conj_physical()
print(A.flatten()[1], torch.tensor(1+1j))
# tensor(-0.2042-0.5812j, grad_fn=&lt;SelectBackward0&gt;) tensor(1.+1.j)
print(A.flatten()[1]*torch.tensor(1+1j))
# tensor(0.3770-0.7854j, grad_fn=&lt;MulBackward0&gt;)
cpp_ext(A)
</code></pre>
<p>c++</p>
<pre><code class="lang-auto">void foo(at::Tensor A) {
    std::cout&lt;&lt;A.accessor&lt;c10::complex&lt;float&gt;,4&gt;()[0][0][0][1]&lt;&lt;c10::complex&lt;float&gt;(1., 1.)&lt;&lt;A.accessor&lt;c10::complex&lt;float&gt;,4&gt;()[0][0][0][1]*c10::complex&lt;float&gt;(1., 1.)&lt;&lt;std::endl;
// (-0.204194,-0.581166)(1,1)(0.376973,-0.78536)
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("cpp_ext", &amp;foo, "ext");
}
</code></pre>
<p>conj<br>
python3:</p>
<pre><code class="lang-auto">A = (...).conj()
print(A.flatten()[1], torch.tensor(1+1j))
# tensor(0.0257+0.6646j, grad_fn=&lt;SelectBackward0&gt;) tensor(1.+1.j)
print(A.flatten()[1]*torch.tensor(1+1j))
# tensor(-0.6389+0.6903j, grad_fn=&lt;MulBackward0&gt;)
cpp_ext(A)
</code></pre>
<p>c++</p>
<pre><code class="lang-auto">void foo(at::Tensor A) {
    std::cout&lt;&lt;A.accessor&lt;c10::complex&lt;float&gt;,4&gt;()[0][0][0][1]&lt;&lt;c10::complex&lt;float&gt;(1., 1.)&lt;&lt;A.accessor&lt;c10::complex&lt;float&gt;,4&gt;()[0][0][0][1]*c10::complex&lt;float&gt;(1., 1.)&lt;&lt;std::endl;
// (0.0257227,-0.664579)(1,1)(0.690302,-0.638856)
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("cpp_ext", &amp;foo, "ext");
}
</code></pre>
Reducing memory cost during `masked_fill`

jasperhyp: <p>Hi!  I am always encountering OOM in this line of <code>MultiheadAttention</code> when training Transformer:</p>
<pre><code class="lang-python">attn_weights = attn_weights.masked_fill(
                key_padding_mask.unsqueeze(1).unsqueeze(2).to(torch.bool), float("-inf")
            )
</code></pre>
<p>Why would this happen?  <code>attn_weights</code>, <code>key_padding_mask</code> are all there already.  If this is indeed taking a lot of memory, how should I reduce the memory cost?  Also, would it improve the memory performance if I use a bunch of <code>del</code>’s and <code>torch.cuda.empty_cache()</code> inside <code>MultiheadAttention</code>.</p>
F.affine_grid in batch mode is slow for high-resolution images

billiout: <p>I’m using <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.affine_grid.html" rel="noopener nofollow ugc"><code>F.affine_grid</code></a> to apply transformation matrices of size <code>Bx2x3</code> to a batch of images <code>Bx3x1024x1024</code>. I noticed that the following code</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn.functional as F

batch_size = 128
resolution = 1024
M = torch.eye(3, device='cuda').unsqueeze(0).expand(batch_size, -1, -1)[:, :2, :]
img = torch.zeros(batch_size, 3, resolution, resolution, device='cuda')

total_time = 0
total_iterations = 20

for _ in range(total_iterations):
    start = time.time()
    grid = F.affine_grid(M, img.shape, align_corners=False)
    transformed_img = F.grid_sample(img, grid, align_corners=False)
    torch.cuda.synchronize()
    total_time += time.time() - start

print(total_time/total_iterations)  # 0.2470783233642578 seconds
</code></pre>
<p>is significantly slower in execution than computing the grid for each batch sample separately</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn.functional as F

batch_size = 128
resolution = 1024
M = torch.eye(3, device='cuda').unsqueeze(0).expand(batch_size, -1, -1)[:, :2, :]
img = torch.zeros(batch_size, 3, resolution, resolution, device='cuda')

total_time = 0
total_iterations = 20

for _ in range(total_iterations):
    start = time.time()
    grid = torch.cat([F.affine_grid(M[i:i+1], [1, *img.shape[1:]], align_corners=False) for i in range(img.shape[0])], dim=0)
    transformed_img = F.grid_sample(img, grid, align_corners=False)
    torch.cuda.synchronize()
    total_time += time.time() - start

print(total_time/total_iterations)  # 0.07158441543579101 seconds
</code></pre>
<p>I believe this is because of a <code>bmm</code> operation in the creation of the grid between <code>Bx1024*1024x3</code> and <code>Bx3x2</code> matrices but still, creating the grid for each batch sample separately seems counterintuitive.</p>
<p>Since my application doesn’t require this operation to be differentiable, I’m wondering if there’s an alternative way of applying a transformation matrix to an image in batch mode that is fast enough for high-resolution images.</p>
How is training optimized

Rhinocamp: <p>I just read about PyTorch 2.0, specifically AOTAutograd and training optimization. I’m curious how exactly is training optimized? Here’s my very high-level speculation:</p>
<ol>
<li>
<code>torch.compile</code> traces the forward and backward computation graph (in some IR format)</li>
<li>at some point, the graph goes through optimizations like fusion (or even lowering to some backend?)</li>
<li>now we can use the result of (2) to run the forward &amp; backward pass, and benefit from the optimizations.</li>
</ol>
<p>And some questions:</p>
<ol>
<li>How does <code>torch.compile(model)</code> trace the graph? I don’t think it has access to the backward pass yet. So does it trace the backward pass lazily?</li>
<li>When is the optimization triggered? Do we cache the optimized graph for the next training iteration?</li>
</ol>
<p>If there’s a better place to post this question, please let me know as well:).</p>

marksaroufim: <p>Your speculation is mostly correct for 1,2 and for your questions optimizations are cached and only redone if some guard assumptions are violated <a href="https://pytorch.org/docs/master/dynamo/guards-overview.html" class="inline-onebox" rel="noopener nofollow ugc">Guards Overview — PyTorch master documentation</a> and for 1 <a class="mention" href="/u/chillee">@chillee</a> will probably explain it better than me</p>

Chillee: <p>We trace out the backwards pass “ahead of time” (hence AOTAutograd). Although we are not actually able to execute the backwards pass until later (when we have gradOut), we know the shape of gradOut, and thus can trace it out “ahead of time”.</p>
<p>You can read more about it here: <a href="https://docs.google.com/presentation/d/1rTt0BR2KChDQQTks2hHUtvHxtHQKwgQHVNrmbhj0byk/edit?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">AOTAutograd - Google Slides</a></p>
Transfer Learning for smaller images

theyellowboots: <p>I am pretty new to Pytorch and I have a question with regards to transfer learning from one of the models downloaded from torchvision.</p>
<p>I understand that the torchvision models were originally trained on images [3 x 224 x 224] in size, but I am looking to adopt one of the models with fewer layers on images [1 x 28 x 28]. The image dataset that I have is very small in size.</p>
<p>What I did firstly was to repeat my first layer 3 times to create a [3 x 28 x 28], and I am using the vgg11 because it is one of the smaller models. However, I am running into the following issue now: <em>Given input size: (512x1x1). Calculated output size: (512x0x0). Output size is too small</em>. My images are way too small for even vgg11.</p>
<p>I understand that there is a way - that is to resize the image a bit bigger to [1 x 128 x 128] and it should work since the newest version of torchvision does allow input tensors that are 224 in height and width. But I believe that expanding images will make my image features more blurry due to the drop in resolution and I think it may impact the model.<br>
Is there a way to trim any model so that it can be suitable for smaller images?</p>

ptrblck: <p>You could copy the <code>torchvision</code> VGG implementation from <a href="https://github.com/pytorch/vision/blob/744e37d7b5dec429b03cda213224543816499cb2/torchvision/models/vgg.py">here</a> and adapt it to your needs (e.g. by removing/replacing layers etc.).<br>
Another approach would be to create the <code>torchvision</code> model object and manipulate the layers directly, but I think this approach would be more suitable for smaller changes, such as replacing the last linear layer for transfer learning.</p>

valtersjz: <p>I had the same problem. Choosing the best solution Id say depends on how small the image is. For (1, 28, 28) I just padded the images to the minimum VGG11 dimensions: (in_channels, 32, 32).</p>
<p>For MNIST:<br>
image_dim = (1,28,28)<br>
So need to add 4 pixels</p>
<p>I used:</p>
<pre><code>import torchvision.transforms as transforms
padding_func = transforms.Pad(2)
x = padding_func(x)
</code></pre>
<p>In most cases probably best to just add it to the transforms:</p>
<pre><code>transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    transforms.Pad(2)
])

test_set = torchvision.datasets.CIFAR10(..., transform=transform)
</code></pre>
Torchmetrics problem, conf matrix and presicion recall acc f1 does not proper

Oguzhan: <p>Hi, the evalation outputs listed below:<br>
Precision:  tensor(0.6233)<br>
Recall:  tensor(0.7200)<br>
F1 score:  tensor(0.6577)<br>
accuracy:  tensor(0.7200)<br>
conf : tensor([[242,   0],<br>
[ 58,   0]])</p>
<p>Is that possible? can be the reason The average method for presicion recall… causes this?</p>
<p>Here is definitions of metrics:</p>
<pre><code class="lang-auto">self.f1 = F1Score(num_class,average = "macro")
self.presicion = Precision(num_class,average = "macro")
self.recall = Recall(num_class, average = "macro")
self.accuracy = Accuracy(num_classes = num_class, average = "macro")
self.conf = ConfusionMatrix(num_classes = num_class,normalize = "none")
</code></pre>
<p>and calculation:</p>
<pre><code class="lang-auto">for data, target in testloader:
            data, target = data, target
            with torch.no_grad():
                output = self(data)
            _,output = torch.max(output,1)
            test_score_conf = test_score_conf + self.conf(output,target)
            test_score_f1 = test_score_f1 + self.f1(output,target)
            test_score_accuracy = test_score_accuracy + self.accuracy(output,target)
            test_score_recall = test_score_recall + self.recall(output,target)
            test_score_presicion = test_score_presicion + self.presicion(output,target)
        
        test_score_f1 = data.shape[0]*test_score_f1/len(test_dataset)
        test_score_accuracy = data.shape[0]*test_score_accuracy/len(test_dataset)
        test_score_recall = data.shape[0]*test_score_recall/len(test_dataset)
        test_score_presicion = data.shape[0]*test_score_presicion/len(test_dataset)
        return test_score_f1,test_score_accuracy,test_score_recall,test_score_presicion,test_score_conf
</code></pre>
PyTorch 2.0 implementation on Hugging Face

LostDude: <p>Is there a way to easily pass the torch.compile directly to Hugging Face’s pipeline? Was thinking of something like this.</p>
<blockquote>
<p>import torch<br>
from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline</p>
<p>tokenizer = AutoTokenizer.from_pretrained(“distilgpt2”)<br>
model = torch.compile(AutoModelForCausalLM.from_pretrained(“distilgpt2”))</p>
<p>generator = pipeline(task=“text-generation”, model=model, tokenizer=tokenizer)</p>
</blockquote>
<p>But was met with this error:</p>
<blockquote>
<p>The model ‘OptimizedModule’ is not supported for text-generation. Supported models are [‘BartForCausalLM’, ‘BertLMHeadModel’, ‘BertGenerationDecoder’, ‘BigBirdForCausalLM’, ‘BigBirdPegasusForCausalLM’, ‘BlenderbotForCausalLM’, ‘BlenderbotSmallForCausalLM’, ‘BloomForCausalLM’, ‘CamembertForCausalLM’, ‘CodeGenForCausalLM’, ‘CTRLLMHeadModel’, ‘Data2VecTextForCausalLM’, ‘ElectraForCausalLM’, ‘ErnieForCausalLM’, ‘GPT2LMHeadModel’, ‘GPTNeoForCausalLM’, ‘GPTNeoXForCausalLM’, ‘GPTNeoXJapaneseForCausalLM’, ‘GPTJForCausalLM’, ‘MarianForCausalLM’, ‘MBartForCausalLM’, ‘MegatronBertForCausalLM’, ‘MvpForCausalLM’, ‘OpenAIGPTLMHeadModel’, ‘OPTForCausalLM’, ‘PegasusForCausalLM’, ‘PLBartForCausalLM’, ‘ProphetNetForCausalLM’, ‘QDQBertLMHeadModel’, ‘ReformerModelWithLMHead’, ‘RemBertForCausalLM’, ‘RobertaForCausalLM’, ‘RoCBertForCausalLM’, ‘RoFormerForCausalLM’, ‘Speech2Text2ForCausalLM’, ‘TransfoXLLMHeadModel’, ‘TrOCRForCausalLM’, ‘XGLMForCausalLM’, ‘XLMWithLMHeadModel’, ‘XLMProphetNetForCausalLM’, ‘XLMRobertaForCausalLM’, ‘XLMRobertaXLForCausalLM’, ‘XLNetLMHeadModel’].</p>
</blockquote>
<p>I could pass the torch.compile function in Hugging Face’s codebase, but that seems like a lot of work for maintenance until Hugging Face implements PyTorch 2.0. Your suggestion is appreciated!</p>

marksaroufim: <p>HuggingFace has an example of how this would work here <a href="https://www.philschmid.de/getting-started-pytorch-2-0-transformers" class="inline-onebox" rel="noopener nofollow ugc">Getting started with Pytorch 2.0 and Hugging Face Transformers</a></p>
Illegal instruction using torchaudio.transforms.Resample wikt Raspberry Pi 4B

astra1234567: <p>Hi !<br>
I am working on the project with PyTorch and Raspberry Pi 4B. When  I run my code on Raspberry, I receive sth like error or message “Illegal instruction”. It is raised in:</p>
<pre><code class="lang-auto">transform = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=new_sample_rate)
tensor = transform(tensor)
</code></pre>
<p>What can be wrong? Obviously, I checked everything on my computer and it works without any errors.<br>
Have you ever seen sth like this?</p>
How to most efficiently select/modify the indices given by .nonzero()?

tueboesen: <p>Suppose I have a tensor x, which is mostly zeros, but does have some elements different from zero.</p>
<p>I can get the indices for those elements by using <code>indices = x.nonzero()</code></p>
<p>However, now I would like to increase the values of all those elements by 1, how do I most efficiently go about doing that?</p>
<p><code>x[indices] += 1</code> does not work.<br>
I can of course do it with a for loop iterating over one index in the indices at a time, but that seems inefficient, and I’m sure that there must be an elegant way of selecting all the indices returned by nonzero() that I just don’t know about?</p>

cojdex: <p>You can use the <code>torch.index_add_()</code> function to efficiently increase the values of the non-zero elements in the tensor by 1.</p>
<pre><code class="lang-auto">import torch

# Create a tensor with some non-zero elements
x = torch.tensor([[1, 0, 0], [0, 2, 0], [0, 0, 3]])

# Get the indices of the non-zero elements
indices = x.nonzero()

# Use torch.index_add_ to increase the values of the non-zero elements by 1
torch.index_add_(x, 0, indices, torch.ones(indices.size(0)))

print(x)  # tensor([[2, 0, 0], [0, 3, 0], [0, 0, 4]])

</code></pre>
<p>The first argument to <code>torch.index_add_</code> is the tensor that you want to modify. The second argument specifies the dimension along which the indices apply. In this case, we are using 0 because the indices apply along the first dimension. The third argument is the tensor of indices, and the fourth argument is the tensor of values to add to the elements at the specified indices.</p>
<p>Using <code>torch.index_add_</code> is much more efficient than using a for loop because it can perform the operation in parallel.</p>
Confusion about hard drive size of torch.save(Dataset) with random_split)

NovoG93: <p>I created a class that inherits from <code>torch.utils.data.Dataset</code> and split it using <code>torch.utils.data.random_split(dataset, [TRAIN_SIZE, TEST_SIZE])</code>.<br>
Afterwards I load it via the <code>torch.utils.data.DataLoader</code> interface and finally save it using torch.save to the hard drive.</p>
<pre><code class="lang-auto">dataset = DataSetClass(df=DATA, transform=transform, device, window=3)

DATASET_LENGTH = len(dataset)
TRAIN_SIZE = int(DATASET_LENGTH * 0.75)
TEST_SIZE = DATASET_LENGTH - TRAIN_SIZE

train_dataset, test_dataset = random_split(dataset, [TRAIN_SIZE, TEST_SIZE])
dataloader_train = DataLoader(dataset=train_dataset, batch_size=1, shuffle=False, num_workers=4)
dataloader_test = DataLoader(dataset=test_dataset, batch_size=1,  shuffle=False, num_workers=4)

torch.save(dataloader_train, "/path/to/train.pt")
torch.save(dataloader_test, "/path/to/test.pt")
</code></pre>
<p>However the saved files are of the same size on the hard drive, why is that?</p>

ptrblck: <p>I think <code>torch.save(dataloader, PATH)</code> would try to pickle and store the actual Python <code>DataLoader</code> object without the internal data, as e.g. your <code>Dataset</code> could load the actual samples lazily, so you might want to store the real <code>DATA</code> object instead of the <code>DataLoader</code> wrapper.</p>

NovoG93: <p>Thank you for your reply!<br>
I will look into that.</p>
Device out of space error when torch.save with multiprocessing in docker

dsicahn: <p>I’m attempting to save a large number of torch Data objects to disk in parallel with the following block</p>
<pre><code class="lang-auto">def to_disk(r):
    path = '/data/protein_data_dir/raw/' + r['ID'] + '.pt'
    g = convert_nx_to_pyg(create_graph(r))
    torch.save(g, path)
    return g

NUM_CORE = 35
with mp.Pool(NUM_CORE) as pool:
    out = list(tqdm(pool.imap_unordered(to_disk, rows), total=len(rows)))
</code></pre>
<p>This will work for some number of iterations and then fail with  “MaybeEncodingError: Error sending result: ‘Data(x=[51, 163], edge_index=[2, 92], edge_attr=[92, 10], y=1.0)’. Reason: ‘RuntimeError(‘unable to write to file &lt;/torch_29804_423365969_110&gt;: No space left on device (28)’)’”. If I then start the loop again it will save some more objects before running into the same error again. Running this on a single process does not induce the error but is slow. There are several terabytes of space on the disk these objects are being saved to. Anyone know whats going on?</p>

ptrblck: <p>Since you are using a multiprocessing pool, I assume you depend on shared memory of your system? If so, you might be running out of shared memory, not actual disk space. Could you check if that’s the case?</p>

dsicahn: <p>Would that be the RAM? The machine has ~512GB RAM and the usage never appears to go above ~50. However, I think the docker container is limited to 64GB of “shared memory” (again not sure on exactly what that means). I could try increasing that maybe.</p>

ptrblck: <p>Yes, usually <code>/dev/shm</code> would use your RAM to allocate the temp. file storage filesystem.<br>
Docker limits it by quite a bit (the default should be 64MB, not GB), so you might want to start your container with <code>--ipc==host</code> (or increase the shared memory usage directly via <code>--shm-size</code>).</p>

dsicahn: <p>Ahh 64MB makes more sense. Its been running for many iterations now so I think that was probably the issue. Thanks!</p>
How to sample from low likelihood region?

Assefa_Seyoum: <p>Given a mean and a covariance matrix, is it possible to get sample with probability less than a threshold (e.g., p &lt; 0.1) in PyTorch?</p>
<p>Something like this</p>
<pre><code class="lang-auto">gaussian = torch.distributions.MultivariateNormal(mean, cov)

samples = gaussian.sample(sample_shape=torch.Size([1000]),
                          tail_prob=0.1)
</code></pre>
<p>Thank you.</p>

KFrank: <p>Hi Assefa!</p>
<aside class="quote no-group" data-username="Assefa_Seyoum" data-post="1" data-topic="167677" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/assefa_seyoum/48/32226_2.png" class="avatar"> Assefa_Seyoum:</div>
<blockquote>
<p>Given a mean and a covariance matrix, is it possible to get sample with probability less than a threshold (e.g., p &lt; 0.1) in PyTorch?</p>
</blockquote>
</aside>
<p>I’m not aware of any built-in functionality of pytorch that will do this.</p>
<p>One sensible approach would be to use some variation of <a href="https://en.wikipedia.org/wiki/Rejection_sampling" rel="noopener nofollow ugc">rejection sampling</a>.</p>
<p>In your case you could sample from your <code>gaussian</code>, compute whether<br>
your sample falls within your low-likelihood <code>tail_prob</code> region, use the<br>
sample if it does, and reject the sample (and resample) if it doesn’t.</p>
<p>For <code>tail_prob = 0.1</code> you will be throwing out 90% of your samples.<br>
However, if this random sampling isn’t the computational bottleneck in<br>
your overall workflow, this 90% inefficiency won’t really matter.</p>
<p>Best.</p>
<p>K. Frank</p>
How to use the model without the GAT module?

aleksandrpikul134: <p>There is such a model (ResNet18 + GAT):</p>
<pre><code class="lang-auto">class AntispoofModel(nn.Module):
    def __init__(self, device="cpu", **kwargs):
        super().__init__()
        resnet = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)
        self.resnet = nn.Sequential(*[i for i in list(resnet.children())[:-2]]).to(device)
        for ch in self.resnet.children():
            for param in ch.parameters():
                param.requires_grad = False
        self.gat = GAT(**kwargs).to(device)
        self.device = device
        self.adj = torch.tensor(grid_to_graph(7, 7, return_as=np.ndarray)).to(device)
        
    def forward(self, x):
        x = self.resnet(x.to(self.device))
        x = x.view(-1, 49, 512)
        #adj = torch.stack([self.adj for i in range(x.shape[0])]).to(self.device)
        x = self.gat(x, self.adj)
        return torch.sigmoid(x)
</code></pre>
<p>I’m trying to do the same thing, but without GAT, for example I rewrite code like:</p>
<pre><code class="lang-auto">class AntispoofModel(nn.Module):    
    
    def __init__(self, device="cpu", **kwargs):
        super().__init__()
        model_resnet = ResNet(BasicBlock, [2, 2, 2, 2])
        self.conv1 = model_resnet.conv1
        self.bn1 = model_resnet.bn1
        self.relu = model_resnet.relu
        self.maxpool = model_resnet.maxpool
        self.layer1 = model_resnet.layer1
        self.layer2 = model_resnet.layer2
        self.layer3 = model_resnet.layer3
        self.layer4 = model_resnet.layer4
        self.avgpool = nn.AdaptiveAvgPool2d(output_size=1)
        
    def forward(self, x):
        feature = self.conv1(x)
        feature = self.bn1(feature)
        feature = self.relu(feature)
        feature = self.maxpool(feature)
        feature = self.layer1(feature)
        feature = self.layer2(feature)
        feature = self.layer3(feature)
        feature = self.layer4(feature)
        out = self.avgpool(feature) #[,512,1,1]
        out = out.view(out.size(0), -1) #[,512]
        return out
</code></pre>
<p>But I have <code>Using a target size (torch.Size([64, 1])) that is different to the input size (torch.Size([64, 512])) is deprecated. Please ensure they have the same size.</code></p>
<p>I understand what it means. But I don’t understand how to fix it…</p>
Intensity classification normalisation

koopa31: <p>Hi everyone,<br>
I am facing a quite simple problem, yet I cannot solve it. I am trying to classify images according to their intensity. Min max normalization or division by max don’t work, as it “kills” the intensity differences. if I don’t normalize at all, it works like charm. The problem is that if I try to process data with lower intensity levels, hence a lower contrast, it only succeeds in classifying the brightest ones.<br>
I am sure there is a way to normalize in a clever way so I do not have to include those new images to my training dataset every time I wanna apply my network to a new dataset. Any idea?<br>
Than you!</p>
Time out while I use dist.barrier()

flying37520: <p><strong>error report:</strong><br>
RuntimeError: CUDA error: the launch timed out and was terminated<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p><strong>brief</strong><br>
I found that <code>dist.barrier()</code> can not wait for long time while use <code>**nccl**</code> backend.</p>
<p>I try to modify like this:<br>
<code>dist.init_process_group(backend, rank=rank, world_size=size,timeout=timedelta(seconds=2000))</code><br>
but it does not work while use nccl backend.<br>
If I use dist.barrier() to pause a process longer than 7 seconds ,above error occur.</p>
<p><strong>code</strong></p>
<pre><code class="lang-auto">def run(rank,size ): 
    idx=0
    while True:
        dist.barrier()
        if rank==0:
            idx+=1
            time.sleep(8) ##Simulation calculation process, while duration time &gt;7s ,Error occur######

        print(f'sync:{idx}')
        dist.barrier()

def init_processes(rank, size ,fn, backend='nccl'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'  # ##
    os.environ['NCCL_ASYNC_ERROR_HANDLING ']='1'
    dist.init_process_group(backend, rank=rank, world_size=size,timeout=timedelta(seconds=2000))
    fn(rank, size)

if __name__ == '__main__':    
    multiprocessing.set_start_method("spawn")
    size = 3  
    processes = []
    for rank in range(size):
        p = Process(target=init_processes, args=(rank, size,run))
        p.start()
        processes.append(p)

    time.sleep(0.05)
    processes.append(p)
    for p in processes:
        p.join()
</code></pre>
Matrix multiplication in torch is slower than in numpy for small matrices

lokesh: <p>torch’s matrix multiplication speed is increasing as the number of multiplications increases for small matrices</p>
<pre><code class="lang-auto">
def timer(func, reps, *args, **kwargs):
    s = perf_counter()
    for i in range(reps):
        func(*args, **kwargs)
    e = perf_counter()

    return e - s



y = list(range(16))
t = torch.Tensor(y * 16).reshape((16, 16))
u = torch.Tensor(y)

v = t.numpy()
w = u.numpy()

n = 8
time_torch_cpu = [timer(torch.matmul, 10**i, t, u) for i in range(1, n)]
time_numpy = [timer(np.dot, 10**i, v, w) for i in range(1, n)]

x = [i for i in range(1, n)]

plt.plot(x, time_numpy, color='red', label='numpy')
plt.plot(x, time_torch_cpu, color='blue', label='pytorch cpu')
plt.ylabel('time (seconds)')
plt.xlabel('No: of multiplications (10^i)')
plt.legend()
plt.show()


</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c290d9dd5eb8c8740f5c648ce6ab9d4f4e8cb0d.png" alt="image" data-base62-sha1="mhsrmUgQLj13vzFVJYt9PErDdpj" width="562" height="432"></p>

ptrblck: <p>This might be expected as PyTorch would most likely add more overhead to its calls due to the Autograd bookkeeping and dispatching.</p>

lokesh: <p>Even when I use torch.inference_mode(), there is only a small improvement. Is there any way to further increase the speed</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e0e0c825cbee81eed1c27d24a7a86354e2c283d.png" alt="image" data-base62-sha1="b8vldBcVf7WSV9wHpMIhqfQBq2x" width="562" height="437"></p>
Why naming `__call__` as `_call_impl` in module.py?

marsggbo: <p>What’s the purpose to name <code>__call__</code> as <code>_call_impl</code> in module.py?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6145ac07b5431a95e7d29f84ab979b332883b4ea/torch/nn/modules/module.py#L1065" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6145ac07b5431a95e7d29f84ab979b332883b4ea/torch/nn/modules/module.py#L1065" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/6145ac07b5431a95e7d29f84ab979b332883b4ea/torch/nn/modules/module.py#L1065</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="1055" style="counter-reset: li-counter 1054 ;">
<li>        grad_fn = var.grad_fn</li>
<li>        if grad_fn is not None:</li>
<li>            for hook in non_full_backward_hooks:</li>
<li>                wrapper = functools.partial(hook, self)</li>
<li>                functools.update_wrapper(wrapper, hook)</li>
<li>                grad_fn.register_hook(wrapper)</li>
<li>            self._maybe_warn_non_full_backward_hook(input, result, grad_fn)</li>
<li>
<li>    return result</li>
<li>
<li class="selected">__call__ : Callable[..., Any] = _call_impl</li>
<li>
<li>def __setstate__(self, state):</li>
<li>    self.__dict__.update(state)</li>
<li>    # Support loading old checkpoints that don't have the following attrs:</li>
<li>    if '_forward_pre_hooks' not in self.__dict__:</li>
<li>        self._forward_pre_hooks = OrderedDict()</li>
<li>    if '_state_dict_hooks' not in self.__dict__:</li>
<li>        self._state_dict_hooks = OrderedDict()</li>
<li>    if '_load_state_dict_pre_hooks' not in self.__dict__:</li>
<li>        self._load_state_dict_pre_hooks = OrderedDict()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


farhanhubble: <p><code>__call__</code> is a magic method in Python. If any object contains an implementation of this method, it can be invoked like a function. In this case, instead of trying to define the <code>__call__</code> method, they opted to define a method <code>_call_impl</code>, suitably named as it is a concrete implementation, and simply set <code>__call__</code> to this method. It’s a design choice and gives some flexibility as in allowing <code>__call__</code> to refer to different implementations based on some condition.</p>
How to normalize custom image dataset

nikiguo93: <p>Hi, can anybody tell me how to normalize the image and make the pixel values between (0,1)?<br>
currently the pixel values of my normalized image data are all between (-1,1).</p>

Karthik_Ganesan: <p>If your input tensor, which is normalized from <code>[-1,1]</code> is <code>x</code>, you should just have to do <code>y = 0.5 + 0.5*x</code> to get a new tensor <code>y</code>, which has the range <code>[0,1]</code>.</p>

srishti-git1110: <p>Hi,<br>
You could also checkout out PyTorch’s <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.Normalize.html#torchvision.transforms.Normalize" rel="noopener nofollow ugc">torchvision.transforms.Normalize</a>.</p>
How torch.cat and torch.split effect the training result?

wutianxu: <p>Hi there,</p>
<p>I am training a network which is modified based on Faster-RCNN. I tried to concat the feature extracted form backbone and split the new feature as follow:</p>
<pre><code class="lang-auto">feat_flatten_cat = torch.cat(feat_flatten, 2) # feat_flatten: list( [BN * channel * WH])
feat_flatten = torch.split(feat_flatten_cat, [feat.shape[-1] for feat in feat_flatten], dim=2)
</code></pre>
<p>I found that the training results were totally different from the case that I comment these two lines of code, the training is very hard to converge and the accuracy is close to 0. Can anyone explain this?</p>
<p>Appreciate any reply.</p>

wutianxu: <p>By the way, for both case, ‘feat_flatten’ is equal, which has been checked by tensor.equal()</p>
Does the "script" method in script module work for nested model (e.g. a resnet)?)

hharryyf: <p>Hi, I’m new to scriptmodule. I’m confused, can the “script” method return a “script module” for a nested model? For example, I have the following resnet defined in python, here InBlock, ResBlock, OutBlock are standard CNN with several layers of conv2d.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.inblock = InBlock()
        self.res_block_1 = ResBlock()
        self.res_block_2 = ResBlock()
        self.res_block_3 = ResBlock()
        self.res_block_4 = ResBlock()
        self.res_block_5 = ResBlock()
        self.res_block_6 = ResBlock()
        self.res_block_7 = ResBlock()
        self.res_block_8 = ResBlock()
        self.outblock = OutBlock()
    
    def forward(self,s):
        s = self.inblock(s)
        s = self.res_block_1(s)
        s = self.res_block_2(s)
        s = self.res_block_3(s)
        s = self.res_block_4(s)
        s = self.res_block_5(s)
        s = self.res_block_6(s)
        s = self.res_block_7(s)
        s = self.res_block_8(s)
        s = self.outblock(s)
        return s
net = Net()
sm = torch.jit.script(net)
sm.save('model.pt')
</code></pre>
<p><strong>Question: I’m confused, 1) does the “script” method work in this situation? 2) If I want to load and train the model with the following C++ code, are ALL the parameters of the model loaded? Or just a subset of the parameters loaded because the “script” method does not work recursively in libtorch?</strong></p>
<pre><code class="lang-auto">auto module = torch::jit::load("model.pt");
        std::vector&lt;torch::jit::IValue&gt; inputs;
        std::vector&lt;at::Tensor&gt; parameters;
        for (const auto&amp; params : module.parameters()) {
	        parameters.push_back(params);
        }
        torch::optim::Adam optimizer(parameters, 0.1);
</code></pre>
<p>Thanks a lot.</p>

ptrblck: <p><code>torch.jit.script</code> should script all properly registered modules recursively and your <code>libtorch</code> export should thus also contain all modules and parameters.</p>
Pytorch 1.13 onnx export is incompatible with TensorRT conversion

Vaidehi_Venkatesan: <p>Pytorch 1.13 APIs for onnx export seem to generate onnx graphs that are incompatible with TensorRT conversion tools we use internally. The bias vectors seem to be added as an input to conv nodes instead of being added as an initializer. For example: Torchvision resnet18 when exported to onnx using torch.onnx.export results in the following graph:</p>
<p>Pytorch 1.13<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f4927e03d4a98b37b032c1010f89261ba3d4f40.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f4927e03d4a98b37b032c1010f89261ba3d4f40" title="Screen Shot 2022-11-30 at 11.09.11 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4927e03d4a98b37b032c1010f89261ba3d4f40_2_373x500.png" alt="Screen Shot 2022-11-30 at 11.09.11 AM" data-base62-sha1="2bdTEdXHTzgsbZIsneoDnQTEp7W" width="373" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4927e03d4a98b37b032c1010f89261ba3d4f40_2_373x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4927e03d4a98b37b032c1010f89261ba3d4f40_2_559x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f4927e03d4a98b37b032c1010f89261ba3d4f40.png 2x" data-dominant-color="E7E6E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-11-30 at 11.09.11 AM</span><span class="informations">704×942 48.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The bias nodes are initialized as part of Conv operator in Pytorch.1.11</p>
<p>Using Pytorch1.13 onnx graph for TensorRT conversion results in the following error:<br>
<code>Errors out with [TRT] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:647 In function importConv: [8] Assertion failed: inputs.at(2).is_weights() &amp;&amp; "The bias tensor is required to be an initializer for the Conv operator."</code></p>
<p>Code to generate onnx graph:</p>
<pre><code class="lang-auto">   import torch
   import torch.onnx as onnx
   import torchvision.models as models
 
   resnet18 = models.resnet18()
   x = torch.randn(1, 3, 224, 224, requires_grad=True)
   onnx_file = "./resnet18.onnx"
   onnx.export(resnet18, x, onnx_file)
</code></pre>
<p>Is this a known regression?</p>

FrederikArvoo: <p>I had the same problem with my network but managed to solve it by using <a href="https://github.com/daquexian/onnx-simplifier" rel="noopener nofollow ugc">onnx simplifier</a>. This removes the extra Identity nodes again, which makes TensorRT able to use the ONNX graph again.</p>

vadimkantorov: <p>I would say, this should rather be reported as bug on <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a></p>
Is there a reason for KL divergence to increase?

banikr: <p>I am using Variational Autoencoder(VAE) and KL divergence with recon loss as criteria.<br>
Interestingly after 100 epochs or so, KL divergence starts to increase exponentially.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99aaa47dd3b92f6b1a472a12f9f2002c1870bf61.png" alt="image" data-base62-sha1="lVoEtV7erWPWpAdlry1oyS6gw13" width="453" height="346"></p>
<p>If I understand correctly shouldn’t it decrease rather than an increase in such a manner?<br>
The code I am using as below:</p>
<pre><code class="lang-auto">    def kl_divergence(z, mu, std):
        p = torch.distributions.Normal(torch.zeros_like(mu), torch.ones_like(std))
        q = torch.distributions.Normal(mu, std)
        log_qzx = q.log_prob(z)
        log_pz = p.log_prob(z)
        kl = (log_qzx - log_pz)
        kl = kl.sum(-1)
        return kl
</code></pre>
<p>where <code>z</code> is 1D spectrum data.</p>

banikr: <p>The same also observed using `torch.distributions.kl_divergence’</p>
<pre><code class="lang-auto">    def kl_divergence(self, z, mu, std):
        p = torch.distributions.Normal(torch.zeros_like(mu), torch.ones_like(std))
        q = torch.distributions.Normal(mu, std)
        kl_t = torch.distributions.kl_divergence(q, p).sum(-1)
        return kl_t
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed3c05cc279429a2160d4535485bea8c995e3d53.png" alt="image" data-base62-sha1="xQFL1nuQ7TjpI6AvfvzNyyOJa71" width="389" height="296"></p>
Low training throughput when using small batch size

ma-sadeghi: <p>I’m using PyTorch to do regression on binary images using a UNet architecture. I’m using Adam optimizer and my training loop is very typical (iterating through a data loader, doing a forward pass, compute the loos, backward pass, and update the weights). In my experience, when I use a small batch size, my training throughput is significantly lower than when using a relatively large batch size.</p>
<p>To give an example to make it more concrete, at a batch size of 256 or greater, the training throughput plateaus at ~500 samples/s. If I use a smaller batch size, let’s say if I halve it, i.e., 128, the throughput halves as well at ~250 samples/s, and this trend almost linearly continues, i.e., at a batch size of 64, I roughly get ~125 samples/s, so on.</p>
<p>In case it’s relevant, my model roughly has 20 million parameters, and my inputs are 2D binary images with a size of 128 by 128. Also, I’m using an RTX 8000 to train the network.</p>
<p>Would greatly appreciate it if you could help min pin down the issue, or if that’s the expected behaviour please explain why.</p>

tiramisuNcustard: <p>I would think in terms of hardware a little. With a powerful card like yours, naturally you would want to fill it (registers, tensor cores etc.) with as much data (images) as possible and then do your binary multiplications etc. For your card, 256 or greater images might be the limit of how much data you can put into the hardware before it saturates (your hardware bottleneck). If you halve the number of images in the batch size then you are not taking the full advantage of all the resources (think not all the registers are full). In the latter case, even if your card is processing the images (multiplications, additions etc.) at full speed - you are not going to go through your entire training batch as fast as possible (your throughput will drop because you lowered the batch size).</p>
<p>Again I am talking purely from the hardware’s perspective.</p>

ma-sadeghi: <p><a class="mention" href="/u/tiramisuncustard">@tiramisuNcustard</a> Thank you so much for your detailed reply. It totally makes sense, and I’m almost sure that this is the case, because when I monitor GPU usage using <code>nvtop</code>, I see almost 100% usage on batch sizes of 256 and beyond, and much less usage (I didn’t exactly monitor the numbers) when using smaller batch sizes.</p>
<p>The only reason I hoped to get a good performance on smaller batch sizes is that in my experience, or better say in my particular case, I get better training performance on smaller batch sizes, given a fixed number of iterations/epochs.</p>

tiramisuNcustard: <p><a class="mention" href="/u/ma-sadeghi">@ma-sadeghi</a>, I agree with you - model performance is different from hardware (GPU) performance. You might get better model performance with different combinations of the hyperparameters (i.e. batch_size, epoch, number of nodes in the hidden layers etc.). Some combinations might give you better performance with larger batch size (a win-win from both the model and hardware performance perspectives).</p>
New backends and structured TensorIterator kernels

tehbone: <p>Hello, I am toying with a backend which cannot make use of a TensorIterator for various reasons. I noticed that if a kernel is structured, I can get away with just implementing the structured delegate like <code>mm.out</code> in the case of <code>mm</code>. That doesn’t seem to be the case if the delegate inherits <code>TensorIteratorBase</code> as in the case of <code>add.Tensor</code> - the codegen invokes the TensorIterator form even though my backend provides an <code>add.out</code>.</p>
<ol>
<li>Is this expected behavior?</li>
<li>If not, is there a codegen means to reduce the boilerplate for external backends? I have looked at what <code>torch_xla</code> does, and it does not seem to use the same structured kernel mechanism (unless I am misinterpreting it).</li>
</ol>

tehbone: <p>I did some digging and I was mistaken. The code <em>should</em> be invoking my implementation of <code>add.out</code>, but it is failing in the <code>meta()</code> function because that depends on the TensorIterator. So I guess my question is whether or not there is a codegen means to bypass TensorIterator use without having to reimplement all of the forms of the operator.</p>