/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/__init__.py:955: UserWarning: torch.set_default_tensor_ty        pe() is deprecated as of PyTorch 2.1, please use torch.set_default_dtype() and torch.set_default_device() as alternatives. (Trigg        ered internally at ../torch/csrc/tensor/python_tensor.cpp:432.)
  _C._set_default_tensor_type(t)
Start loading the data....
  - Found cached train data
/share/home/heruliang/Husformer/src/utils.py:16: FutureWarning: You are using `torch.load` with `weights_only=False` (the current         default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will ex        ecute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more de        tails). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could         be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly         allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any u        se case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this exper        imental feature.
  - Found cached valid data
  - Found cached test data
Finish loading the data....
/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:60: UserWarning: The verbose parame        ter is deprecated. Please use get_last_lr() to access the learning rate.
  warnings.warn(
/share/home/heruliang/Husformer/modules/position_embedding.py:20: UserWarning: The number of elements in the out tensor of shape         [700] is 700 which does not match the computed number of elements 1468. Note that this may occur as a result of rounding error. T        he out tensor will be resized to a tensor of shape (1468,). (Triggered internally at ../aten/src/ATen/native/cuda/RangeFactories.        cu:250.)
  torch.arange(padding_idx + 1, max_pos, out=getattr(make_positions, buf_name))
Traceback (most recent call last):
  File "/share/home/heruliang/Husformer/main-4.py", line 132, in <module>
    test_loss = train.initiate(hyp_params, train_loader, valid_loader, test_loader)
  File "/share/home/heruliang/Husformer/src/train.py", line 29, in initiate
    return train_model(settings, hyp_params, train_loader, valid_loader, test_loader)
  File "/share/home/heruliang/Husformer/src/train.py", line 151, in train_model
    _, mae_train = train(model, optimizer, criterion)
  File "/share/home/heruliang/Husformer/src/train.py", line 84, in train
    preds, hiddens, _, _, _, _ = net(m1, m2, m3, m4)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_cal        l_impl
    return self._call_impl(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 186, in forwa        rd
    outputs = self.parallel_apply(replicas, inputs, module_kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 201, in paral        lel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 109, in para        llel_apply
    output.reraise()
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/_utils.py", line 706, in reraise
    raise exception
torch.OutOfMemoryError: Caught OutOfMemoryError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 84, in _work        er
    output = module(*input, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_cal        l_impl
    return self._call_impl(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/share/home/heruliang/Husformer/src/models.py", line 94, in forward
    m3_with_all = self.trans_m3_all(proj_x_m3, proj_all, proj_all)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_cal        l_impl
    return self._call_impl(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/share/home/heruliang/Husformer/modules/transformer.py", line 81, in forward
    x = layer(x, x_k, x_v)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_cal        l_impl
    return self._call_impl(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/share/home/heruliang/Husformer/modules/transformer.py", line 150, in forward
    x, _ = self.self_attn(query=x, key=x_k, value=x_v, attn_mask=mask)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_cal        l_impl
    return self._call_impl(*args, **kwargs)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/share/home/heruliang/Husformer/modules/multihead_attention.py", line 118, in forward
    attn_weights = F.dropout(attn_weights, p=self.attn_dropout, training=self.training)
  File "/share/home/heruliang/conda/envs/hus/lib/python3.10/site-packages/torch/nn/functional.py", line 1295, in dropout
    return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.47 GiB. GPU 0 has a total capacity of 31.73 GiB of which 330.94 M        iB is free. Including non-PyTorch memory, this process has 31.40 GiB memory in use. Of the allocated memory 29.10 GiB is allocate        d by PyTorch, and 1.41 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORC        H_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org        /docs/stable/notes/cuda.html#environment-variables)

这个报错怎么解决