Memory error when installing pytorch

username: garytho
cooked: <p>Hello,</p>
<p>I’ve had trouble installing pytorch locally on a shared computing cluster.</p>
<p>When I tried to install pytorch in a python 3.6 virtualenv with pip3 I got the following error:</p>
<blockquote>
<p>Exception:<br>
Traceback (most recent call last):<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/basecommand.py”, line 215, in main<br>
status = self.run(options, args)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/commands/install.py”, line 335, in run<br>
wb.build(autobuilding=True)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/wheel.py”, line 749, in build<br>
self.requirement_set.prepare_files(self.finder)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/req/req_set.py”, line 380, in prepare_files<br>
ignore_dependencies=self.ignore_dependencies))<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/req/req_set.py”, line 620, in _prepare_file<br>
session=self.session, hashes=hashes)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/download.py”, line 821, in unpack_url<br>
hashes=hashes<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/download.py”, line 663, in unpack_http_url<br>
unpack_file(from_path, location, content_type, link)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/utils/<strong>init</strong>.py”, line 599, in unpack_file<br>
flatten=not filename.endswith(‘.whl’)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/site-packages/pip/utils/<strong>init</strong>.py”, line 488, in unzip_file<br>
data = zip.read(name)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/zipfile.py”, line 1309, in read<br>
return fp.read()<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/zipfile.py”, line 833, in read<br>
buf += self._read1(self.MAX_N)<br>
File “/apps/python3/python3-3.6.1-ic-2017-mkl2/lib/python3.6/zipfile.py”, line 923, in _read1<br>
data = self._decompressor.decompress(data, n)<br>
MemoryError</p>
</blockquote>
<p>I retried this using a compute node with 5.05 GB of memory and I got the same error.</p>
<p>I tried to install pytorch on conda with the command (conda install pytorch torchvision cuda80 -c soumith), but I got this error:</p>
<blockquote>
<p>ERROR conda.core.link:_execute_actions(337): An error occurred while installing package ‘soumith::pytorch-0.2.0-py36h53baedd_4cu80’.<br>
MemoryError()<br>
Attempting to roll back.<br>
MemoryError()</p>
</blockquote>
<p>How can I install pytorch?</p>
<p>Thank you for your time,</p>
<p>Gary</p>

username: FuriouslyCurious
cooked: <p>Test your install inside Anaconda Docker container first.</p>
<p><a href="https://hub.docker.com/r/continuumio/anaconda3/" class="onebox" target="_blank" rel="nofollow noopener">https://hub.docker.com/r/continuumio/anaconda3/</a></p>
<p>Does it work well there?</p>

username: sivanand_achanta
cooked: <p>Hi,<br>
I too have encountered a similar problem. Could you let me know how you resolved this issue if and when it gets done plz …</p>

username: garytho
cooked: <p>You probably don’t have enough RAM.</p>
<p>I was allocating memory incorrectly on the shared computation cluster, which is why I got the memory error.</p>

username: fovwin
cooked: <p>I solved the issue by adding swap and <code>--no-cache-dir</code><br>
(My ram’s size is 512mb)</p>
<p>first: create swap</p>
<pre><code class="lang-auto"># create swap file of 512 MB
dd if=/dev/zero of=/swapfile bs=1024 count=524288
# modify permissions
chown root:root /swapfile
chmod 0600 /swapfile
# setup swap area
mkswap /swapfile
# turn swap on
swapon /swapfile
</code></pre>
<p>second: install without cache dir</p>
<pre><code class="lang-auto">pip3  --no-cache-dir  install http://download.pytorch.org/whl/cu75/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
</code></pre>
<p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/29466663/memory-error-while-using-pip-install-matplotlib" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4753510/nishanth-anand" target="_blank" rel="nofollow noopener">
    <img alt="nishanth anand" src="https://lh3.googleusercontent.com/-1DBs7xyz6P8/AAAAAAAAAAI/AAAAAAAAAoU/wFnF00WtL14/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/29466663/memory-error-while-using-pip-install-matplotlib" target="_blank" rel="nofollow noopener">Memory error while using pip install Matplotlib</a>
</h4>

<div class="tags">
  <strong>python, matplotlib</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4753510/nishanth-anand" target="_blank" rel="nofollow noopener">
    nishanth anand
  </a>
  on <a href="https://stackoverflow.com/questions/29466663/memory-error-while-using-pip-install-matplotlib" target="_blank" rel="nofollow noopener">06:47AM - 06 Apr 15</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://s2.wp.com/i/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://chirale.org/2017/01/15/memory-error-on-pip-install-solved/" target="_blank" title="08:25AM - 15 January 2017" rel="nofollow noopener">chirale – 15 Jan 17</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:440/248;"><img src="https://chirale.files.wordpress.com/2017/01/memory-error-pip.png?fit=440%2C330" class="thumbnail"></div>

<h3><a href="https://chirale.org/2017/01/15/memory-error-on-pip-install-solved/" target="_blank" rel="nofollow noopener">Memory Error on pip install (SOLVED)</a></h3>

<p>Memory Error when using pip install on Python can emerge both from command line or from a IDE like PyCharm, usually when the package size is big. When you try to install a python package with pip i…</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: lalit
cooked: <p>yes i got the same error while installing pytorch on my VM with pip package and i allocate 1gb ram to VM…please suggest me how can i overcome it</p>

username: mathieucaroff
cooked: <p>In my case, just using the pip option <code>--no-cache-dir</code> was sufficient to make the install work.</p>
<hr>
<p>About my config, I have 1Mo of RAM on my AWS EC2 debian server. Here are the outputs of <code>lsb_release</code> and <code>free</code>, after the installation of torch:</p>
<pre><code class="lang-auto">admin@server:~$ lsb_release -cds
Debian GNU/Linux 10 (buster)
buster
admin@server:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          987Mi       439Mi       246Mi        10Mi       301Mi       396Mi
Swap:            0B          0B          0B
</code></pre>

username: harshrathod50
cooked: <p>In my case, on Fedora 23 <strong>VPS</strong>, I used the given below hack:</p>
<pre><code class="lang-auto">mkdir /tmp2
export TMPDIR=/tmp2
python3.8 -m pip install --no-clean --no-cache-dir torch torchvision
</code></pre>