akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
"Destroy" TCPStore?

username: fot
cooked: <p>Hi all!</p>
<p>I was wondering if there is a way to destroy a TCPStore, and close sockets associated with it.<br>
I want to destroy the current process group (created with a TCPStore instance), and initialize a new one at the same port, but, as expected, I get an “Address already in use” error.</p>
<p>Thank you!</p>

username: wanchaol
cooked: <p>Thanks for posting the question <a class="mention" href="/u/fot">@fot</a>, Internally when constructing the TPCStore, we have a TCPStoreWorkerDaemon to bind the ports and do create creation, so when it goes out of scope and no reference to the TCPStore created, we will call the destructor automatically and release the port resources.</p>
<p>Could you elaborate more on how you get the error? If you did Ctrl-C on one process, there might be hanging processes still have TCPStore as it does not have master to send the destruction signals. In this case you might need to manually kill those hanging processes. You can do it by inspecting what ports your training_script binded to by using <code>$(ps aux | grep training_script.py | grep -v grep | awk '{print $2}') </code>, then kill them</p>

username: wanchaol
cooked: <p>BTW we do have a <code>destroy_process_group(group=None)</code> API if you want to do it programmatically</p>

username: fot
cooked: <p>Thank you for your answer! I would ideally like to destroy the TCPStore and TCPStoreWorkerDaemon without killing the processes that listen to the specific port. I was wondering if this is possible from the Python API. “destroy_process_group(group=None)” does not exactly do what I want, as the TCPStore is not removed from what I understand.</p>

username: wanchaol
cooked: <p>yeah just filed a issue about this, we don’t have a destructor or API that could call to release those ports now, tracking it here <a href="https://github.com/pytorch/pytorch/issues/72025" class="inline-onebox" rel="noopener nofollow ugc">[c10d] destruction of Store objects · Issue #72025 · pytorch/pytorch · GitHub</a></p>

username: fot
cooked: <p>Ok, great, thank you!</p>
Calling function in main()

username: KanZa
cooked: <pre><code class="lang-auto">from train_test import test 
 
class Trainer():
  def __init__(self, method, data_unlabeled, subset, labeled_set, cycle, args):
        self.method = method 
        self.data_unlabeled = data_unlabeled  
        self.subset = subset 
        self.labeled_set = labeled_set 
        self.cycle = cycle 
        self.args = args 

     def query_samples(self, method, data_unlabeled, subset, labeled_set, cycle, args,  param):
        .......... 
        optimizer_target = getattr(optim, param['optimizer'])(net.parameters(), lr = 
        param['learning_rate'])
     
     self.query= query_samples
   
  @classmethod
  def objective(self,train):
     
     params = {}
     
     model = Trainer()
     model = model.query(params)
     accuracy = model.test(trial)
     return accuracy 

if __name__ == '__main__':

    study = optuna.create_study(direction="maximize")
    study.optimize(lambda trial: Trainer.objective(trial))
</code></pre>
<p>init__() missing 6 required positional arguments: ‘method’, ‘data_unlabeled’, ‘subset’, ‘labeled_set’, ‘cycle’, and ‘args’</p>
<p>how to pass these arguments in Trainer class and then call query_samples or query()</p>
<p>If I take out the objective() from Trainer class, I am unable to call the objective function in my main() because of self</p>
<p>File “main.py”, line 94, study.optimize(lambda trial: objective(trial)) TypeError: objective() missing 1 required positional argument: ‘trial’</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/kanza">@KanZa</a></p>
<ul>
<li>you are initializing Trainer object inside a function of the class</li>
</ul>
VAE loss function super negative training loss

username: eric_zhu
cooked: <p>I coded my loss function following <a href="https://arxiv.org/pdf/1907.08956v1.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1907.08956v1.pdf</a>, but the training loss is extremely negative. Specifically, I’m using the negative of the log-likelihood function with MSE being my reconstruction loss. Why might this be?</p>
<p>My code for the loss function is:</p>
<pre><code class="lang-python">MSELoss_criterion = nn.MSELoss()
MSE_loss = MSELoss_criterion(y_hat, tgts) 

KLDiv_loss = -0.5*torch.sum(1+log_var_q - mu_q **2 - log_var_q.exp(), dim=(2)) 
KLDiv_loss = torch.mean(KLDiv_loss) 
return -MSE_loss + KLDiv_Loss
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/eric_zhu">@eric_zhu</a> If you allow negative sign on MSELoss your model will have difficulty converging, as MSME is always positive and to reduce the loss, it will just keep making the variables larger and larger, which is why you are seeing extremely negative loss</p>
<p>I dont think you can use MSME loss as a replacement for the ELBO loss mentioned in the paper</p>
How to make a Tensor from ndarray of type object

username: Muhammad_Saqlain
cooked: <p>I am new to PyTorch. I have an array of length 6 and shape (6, ) when I run torch.from_numpy(data_array), I got this error:</p>
<p>TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.</p>
<p>I have also tried with pd.DataFrame, but face another error:</p>
<p>TypeError: expected np.ndarray (got DataFrame)</p>
<pre><code class="lang-auto">data_array = np.load('Dance0368.npy', allow_pickle=True)
data = pd.DataFrame(data_array)
len(data_array)
6

data_array.shape
(6,)

torch.from_numpy(data_array)
TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

data.shape
(6, 1)

data.dtypes
0    object
dtype: object

data
	0
0	{'cam': [0.371, -0.4058, 0.0315], 'pose': [-2....
1	{'cam': [0.3848, 0.198, 0.03775], 'pose': [-3....
2	{'cam': [0.5635, 0.684, 0.1471], 'pose': [-3.0...
3	{'cam': [0.3994, 0.4841, 0.01412], 'pose': [-3...
4	{'cam': [0.3486, -0.0975, 0.03378], 'pose': [3...
5	{'cam': [0.401, -0.661, -0.0003161], 'pose': [...

torch.from_numpy(data)
TypeError: expected np.ndarray (got DataFrame)

X = torch.from_numpy(data.values)
TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

torch.Tensor(data)
ValueError: expected sequence of length 6 at dim 0 (got 1)
</code></pre>

username: knoriy
cooked: <p>Hi Muhammad,</p>
<p>The problem is caused by the data type you are trying to convert into a <code>torch.Tensor,</code> PyTorch doesn’t support complex data structures such as dictionaries.</p>
<p>You can convert your data into PyTorch tensors by:</p>
<pre><code class="lang-auto">data_array = np.array({"cam":[0,1,1], "pos":[0.5,1.3,1.9]})
cam, pos = torch.tensor([data_array.item()["cam"], data_array.item()["pos"]])
</code></pre>
<p>or</p>
<pre><code class="lang-auto">tensorData= torch.tensor([data_array.item()["cam"], data_array.item()["pos"]])
</code></pre>

username: bobey
cooked: <p>Convert the data to list instead of ndnumpy</p>
Cudnn uses more GPU memory and takes longer to train

username: maxwxzheng
cooked: <p>Hi! I’ve been testing resource usage of my pytorch model with and without cudnn. It seems when I have torch.backends.cudnn.enabled=False, training the network is 2 to 3 times faster and uses less memory (bigger batch size). I tested the same network on Nvidia 3090, 2080 ti and 1080 ti. The result is pretty consistent.<br>
I thought cudnn was supposed to improve the performance of training. Is there anything I can do to investigate this further?<br>
I use conda to manage my environment. Here are the specs:<br>
Ubuntu 18.04<br>
CUDA 11.1<br>
Nvidia driver 455.32<br>
Pytorch 1.8.1<br>
Cudnn 8.0.5_0</p>

username: ptrblck
cooked: <p>Yes, cudnn should accelerate the workload. Could you post the model definition so that we could take a look at it?</p>

username: maxwxzheng
cooked: <p>The exact architecture is a bit complicated. In a nutshell, it’s an encoder-decoder architecture with Efficientnet as the encoder. Is this information helpful?</p>

username: ptrblck
cooked: <p>Unfortunately not, since we are seeing speedups in cudnn for Efficientnet.</p>

username: maxwxzheng
cooked: <p>We use Efficientnet as the backbone and FPN as the decoder for semantic segmentation. We take outputs from different stages in Efficientnet and feed them to FPN in different layers. Does this help?</p>

username: jinluyang
cooked: <p>I guess “cudnn taking more GPU memory” is because cudnn needs workspaces, and if we set cudnn.benchmark=True, it will allocate the max workspace for each algorithms, which will cost more mems. I have this trouble now which often gets OOM error using cudnn.</p>
Transformer decoder training. Batch slice

username: jubick
cooked: <p>So for the transformer decoder we want to feed embedded tokens but not EOS tokens.<br>
Many tutorials do it like x[:,:-1] or x[:-1,:] (BxL or LxB B-batch size N-sequence length)<br>
It makes sense for one sentence “BOS Bob is cool EOS” → “BOS Bob is cool” as we don’t want anything to be predicted after last passed token.<br>
But in batch this isn’t always true.<br>
I don’t understand how to do this slice in batch where not all of the sequences got same length.<br>
For example “BOS Bob is cool EOS” → “BOS Bob is cool” but “BOS Need help EOS PAD” → “BOS Need help EOS” if we simply slice last element.<br>
Surely I’ve added ignore_index to loss but I’m not sure if that’s all we need.<br>
Any help appreciated!</p>

username: jimmykobe
cooked: <p>May I ask have you solved this issue?</p>
.hal() or the use of mixed precision increases model size

username: sardanian
cooked: <p>When I apply .half() to my model it seems to increase the GPU memory uses when I am looking at it in nvidia-smi. It goes from ~4.5GB to ~6GB</p>
<p>I was under the impression that when I apply .half() that it would shrink my GPU usage allowing me to run more in parallel or at the same time.</p>
<p>I was running some tests to see if .half() would speed up my model (and it does) but it increases GPU memory in both single processing or multiprocessing modes.</p>
<blockquote>
<p>def loadModel(model, multiProcessFlag=True, PATH=BASE_MODEL_NAME, half=False):<br>
if multiProcessFlag:<br>
mp.set_start_method(“spawn”, force=True)<br>
model = Net()<br>
model.share_memory()<br>
model.load_state_dict(torch.load(PATH))<br>
model.eval()<br>
model.to(device)<br>
if half:<br>
model.half()<br>
return model</p>
</blockquote>
<p>This also increases the model size when I run inference on it using Mixed Precision (if I do not use half())</p>
<p>So the model will load as the smaller model, but when inference is run the first time on it with Mixed Precision (by running “with autocast” ) then the GPU memory increases as well.</p>

username: sardanian
cooked: <p>So I have to save the model after calling .half()</p>
<p>Then when I reload the new model it will be smaller.</p>
Why are the mathematical formulas and source code different in CosineAnnealingLR?

username: JHLee
cooked: <p>Hello.<br>
When looking at the source code of get_lr method in CosineAnnealing,<br>
I wonder why the return value  contains  “(1 + math.cos(math.pi * (self.last_epoch - 1) / self.T_max))” instead of 0.5.</p>
<blockquote>
<pre><code>    return [(1 + math.cos(math.pi * self.last_epoch / self.T_max)) /
</code></pre>
</blockquote>
<pre><code>            (1 + math.cos(math.pi * (self.last_epoch - 1) / self.T_max)) *
            (group['lr'] - self.eta_min) + self.eta_min
            for group in self.optimizer.param_groups]
</code></pre>
<p>according to mathematical formula in the docs, when Tcur != (2k+1)Tmax<br>
I think it should be multiplied by 0.5.</p>