Issue 1:
🐛 Bug

Hello,

I'm having a problem of loading a serialized tensor from a file.

My tensor shape is [309000001, 2, 5] the dtype is torch.int8

When I deserialize the tensor using torch.load(), it yell "invalid memory size".

The line that it complain is:

pytorch/torch/serialization.py

Lines 528 to 529
in
b740b92

deserialized_objects[root_key] = restore_location(

data_type(size), location)

It seem like the variable it used is destructured from here:

pytorch/torch/serialization.py

Line 525
in
b740b92

data_type, root_key, location, size, view_metadata = data

Which also came from here:

pytorch/torch/serialization.py

Line 517
in
b740b92

data = saved_id[1:]

So I insert print(saved_id) where it yell:

Now, it print:

('storage', <class 'torch.CharStorage'>, '2149339138576', 'cpu', -1204967286, None)

It seem that size is negative.

To Reproduce

import torch

torch.save(torch.zeros([309237982, 2, 5], dtype=torch.int8), './sample.pt')
torch.load('./sample.pt')

It can only be reproduce in MS.Windows environment.

My Windows version is:

10.0.17134.472

My Python version is:

3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)]

Steps to reproduce the behavior:

Run above code in MS. Windows

RuntimeError Traceback (most recent call last)

in

2

3 torch.save(torch.zeros([309237982, 2, 5], dtype=torch.int8), './sample.pt')

----> 4 torch.load('./sample.pt')

c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module)

365 f = open(f, 'rb')

366 try:

--> 367 return _load(f, map_location, pickle_module)

368 finally:

369 if new_fd:

c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module)

538 unpickler = pickle_module.Unpickler(f)

539 unpickler.persistent_load = persistent_load

--> 540 result = unpickler.load()

541

542 deserialized_storage_keys = pickle_module.load(f)

c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in persistent_load(saved_id)

504 print(data_type, size)

505 deserialized_objects[root_key] = restore_location(

--> 506 data_type(size), location)

507 storage = deserialized_objects[root_key]

508 if view_metadata is not None:

RuntimeError: $ Torch: invalid memory size -- maybe an overflow? at ..\aten\src\TH\THGeneral.cpp:188

Expected behavior

Tensor is loaded without error.

Environment

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch Version (e.g., 1.0): 1.0

OS (e.g., Linux): Windows 10

How you installed PyTorch (conda, pip, source): pip

Build command you used (if compiling from source):

Python version: 3.6.7

CUDA/cuDNN version: N/A

GPU models and configuration: N/A

Any other relevant information: No problem on Linux

Additional context

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm

Issue 2:
Issue description

From this post, the following error will be thrown when the input size exceeds 4GB.

Traceback (most recent call last):
File "train.py", line 159, in <module>
main()
File "train.py", line 111, in main
for i_batch, sample_batched in enumerate(Dataloader_Train):
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
OverflowError: cannot serialize a bytes object larger than 4 GiB

(py3_pt3_orig) C:\Users\TestUser\Documents\eliaseulig\DeepDSA>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 116, in _main
self = pickle.load(from_parent)
EOFError: Ran out of input

Code example

Please try to provide a minimal example to repro the bug.

Error messages and stack traces are also helpful.