How to randomly segment lines from one text file into two different text files

username: krishna511
cooked: <p>I have a text file with say, 100 lines, I want to randomly segment these lines into 80-20.Using the code below but its not doing proper partition. I am getting a different number of files . I should get 80 lines in file2 and 20 files in file1.<br>
<strong>can someone point out the error and plz suggest if there is a better way</strong>. please note in total.txt is the original file which needs to be segmented into file1 and file 2.</p>
<p>Regards</p>
<pre><code class="lang-auto">def partition(l, pred):
    fid_train=open('meta/file1.txt','w')
    fid_test = open('meta/file2.txt','w')
    for e in l:
        if pred(e):
            fid_test.write(e)
        else:
            #fid_train.write(e+'\n')
            fid_train.write(e)
    return fid_train,fid_test
lines = open("meta/total_list.txt").readlines()
lines1, lines2 = partition(lines, lambda x: random.random() &lt; 0.2)        
</code></pre>

username: ptrblck
cooked: <p>Your code works correctly using <code>lists</code>:</p>
<pre><code class="lang-python">def partition(l, pred):
    fid_train = []
    fid_test = []
    for e in l:
        if pred(e):
            fid_test.append(e)
        else:
            fid_train.append(e)
    return fid_train,fid_test

lines = ['{}\n'.format(i) for i in range(100)]
lines1, lines2 = partition(lines, lambda x: random.random() &lt; 0.2)   
print(len(lines1))
&gt; 82
print(len(lines2))
&gt; 18
</code></pre>
<p>and I’m unsure why the files should fail assuming <code>readlines()</code> returns the expected number of lines in the source file.<br>
Note that you are not closing the files, which is not recommended.</p>

username: krishna511
cooked: <p>yes sir, <a class="mention" href="/u/ptrblck">@ptrblck</a> in lists its working but its giving only numbers, I need to read these lines. The total_list has the contents as<br>
raw_data\Ses01F_impro01_M013.wav 1 1<br>
raw_data\Ses01F_impro02_F000.wav 2 1<br>
raw_data\Ses01F_impro02_F001.wav 2 1<br>
raw_data\Ses01F_impro02_F002.wav 2 1<br>
raw_data\Ses01F_impro02_F003.wav 3 1</p>
<p>Now I need two separate text files which have a random selection of these files as 80- 20 %<br>
, My code is doing so but the selection is not exact division,<br>
I hope I am clear now.</p>
<ul>
<li>
</li>
</ul>

username: ptrblck
cooked: <aside class="quote no-group" data-username="krishna511" data-post="3" data-topic="119609">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna511/48/30486_2.png" class="avatar"> krishna511:</div>
<blockquote>
<p>My code is doing so but the selection is not exact division</p>
</blockquote>
</aside>
<p>That would be expected, since you are randomly splitting it.<br>
If you want an exact division in 80%-20%, you could use something like:</p>
<pre data-code-wrap="python"><code class="lang-python">lines = [a for a in range(100)]
idx = torch.randperm(len(lines))
idx_train = idx[:int(0.8*len(idx))]
idx_val = idx[int(0.8*len(idx)):]
</code></pre>

username: krishna511
cooked: <p>Sir I have tried a similar thing initially, But then how to write these two files using indexes?<br>
The source file named ‘total_list.txt’ is created in same script. I have closed it,<br>
Then open it, the train val ids are there but from source file I need to extract those lines with corresponding Idx into two text files. I have tried this</p>
<pre><code class="lang-auto">fid_train=open('meta/training1.txt','w')
fid_test = open('meta/testing1.txt','w')
f1=open("meta/total_list.txt").readlines()
lines = [a for a in range(100)]
idx = torch.randperm(len(lines))
idx_train = idx[:int(0.8*len(idx))]
idx_val = idx[int(0.8*len(idx)):]

for lines in f1:
    fid_test.write(f1[idx_val])
    
fid_test.close()

for lines in f1:
    fid_train.write(f1[idx_train])
    
fid_train.close()    
</code></pre>
<p>but it throws error :only integer tensors of a single element can be converted to an index<br>
Also replaced f1 with lines ,</p>
<pre><code class="lang-auto">for lines in f1:
    fid_test.write(lines[idx_val])
    
fid_test.close()

</code></pre>
<p>I am not getting this error actually why it says only integer tensors … all indexes are integer only.<br>
Thanks for reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: krishna511
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  sir the problem is solved,</p>
<pre><code class="lang-auto">lines = open("meta/total_list.txt").readlines()
random.shuffle(lines)

open('meta/training1.txt', 'w').writelines(lines[:80])
open('meta/testing1.txt', 'w').writelines(lines[80:])
</code></pre>
<p>Regards</p>