opened
annashcherbina
username_0
Dear authors, 

I am very excited to use GaKCo-SVM, thank you for making this code available. 
I am able to execute the provided examples without issue, and am able to run on small fragments of my own data. However, when i attempt to train on a larger dataset, I am getting a segmentation fault. The command I am executing is:

```
GaKCo heterokaryon.M5.train.fa dict.txt heterokaryon.M5.train.labels.txt 7 5 heterokaryon.M5.kernel 1
```

The output is: 
![image](https://user-images.githubusercontent.com/5261545/32511525-d872785a-c3a8-11e7-95d0-9c986fa5592d.png)

Can you please advise what the issue could be? 
This is running on ubuntu server 16.04. 

Thank you,
Anna 

Let me know if you'd like me to upload the files -- they are a bit large (~ 1 million fasta regions, each 2kb) so I thought I'd ask before uploading.
Segmentation fault when running on a large dataset
issue
created
rs3zz
username_1
Hello Anna,

We are very excited that you are using GaKCo-SVM. Thank you.
The memory segment issue you are running into is because of ~ 1 million
fasta regions since the code
is running out of memory on your computer for a 1million x 1million kernel
matrix. This will be an issue unless you have a very large memory
size.

We would recommend that you uniformly sample ~20,000-30,000 sequences and
generate the kernels. If you still run into a segfault, then please
let us know along with the memory capacity (4GB/8GB/16GB/64GB) of your
machine.
comment
created
annashcherbina
username_0
Thanks for the quick response! 
The server has 256 GB of ram. 
I am able to use file "thisworks.fa" (15000 sequences, each 2kb) to generate a kernel.
Adding 1000 sequences (also 2kb each) produces a segfault (see "toobig.fa") 
I am a little surprised that with 256GB ram I can only use 15000 sequences. Do these numbers make sense given your experience?  I will follow your advise of random sampling, just wanted to make sure that my numbers are not way off from what you would expect. 
Thanks again for your help! 
Anna 
[toobig.fa.gz](https://github.com/QData/GaKCo-SVM/files/1452404/toobig.fa.gz)
[thisworks.fa.gz](https://github.com/QData/GaKCo-SVM/files/1452405/thisworks.fa.gz)
comment
created
annashcherbina
username_0
I have access to a server w/ 512 GB of RAM, can benchmark if you like?
comment
created
rs3zz
username_1
Hello Anna,

I think I figured out what the issue is. In shared.cpp file, we have
defined MAXNSTR (maximum number of strings) to 15000. That is why the code
breaks after 15k strings.  If you change that number to more than that, and
with enough memory, the code should run. The maximum number of samples we
have tried during our experiments is ~20000 strings, so the code should run
for at least that number. Therefore, feel free to let us know if and when
the code is breaking before or beyond that.

We would definitely work to fix any memory leaks we may have missed. Thanks
again for reaching out.

Regards
Ritambhara
comment
closed
annashcherbina
username_0
issue