Question: I am trying to pull regions from a bunch of BAM files on an online server. I'd like to pull the reads mapping to a certain 1kb or so chunk and download them for analysis. They are far too massive to download them all, and it's impractical even to wget them one at a time and pull the regions out using samtools (tried it, and it worked, but it took forever). Since I'll have to do this for a number of regions that I won't know in advance, I need a better way.

I noticed that samtools is capable of running 'samtools view' off of a web address. Sadly, this data is protected behind an https server, which samtools doesn't know how to handle. I notice that IGV is able to read the BAM files of the net by asking for my login and querying specific regions only that I bring up to view, but I don't have a way of automating the process on hundreds of files.

Does anyone have any ideas of how to run something like samtools view on specific regions over an https connection?

Answer: Possibly can be done through `curl`. In the following command, `--negotiate` option enables SPNEGO in `curl`. The `-u` option is required but the user name is ignored. The `-b` and `-c` options are used to store and send HTTP cookies. The `-s` is to silence `curl `chucking out status. (Try typing the command as it is as the bam file exists)

curl --negotiate -u : -b ~/cookienumnumnum.txt -c ~/cookienumnumnum.txt -s http://gasv.googlecode.com/files/Example.bam | samtools view -h - | head

will give

@SQ SN:chr17 LN:78774742
chr17_15_201_1:0:0_1:0:0_209a6 163 chr17 15 60 50M = 152 187 GTTCCTGCATAGATAATTGCATGACAATTGCCTTGTCCCTCCTGAATGTG 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:37 AM:i:23 X0:i:1 X1:i:0 XM:i:1 XO:i:0 XG:i:0 MD:Z:40G9
chr17_45_242_0:0:0_1:0:0_50f5f 99 chr17 45 60 50M = 193 198 CCTTGTCCCTGCTGAATGTGCTCTGGGGTCTCTGGGGTCTCACCCACGAC 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:0 SM:i:37 AM:i:37 X0:i:1 X1:i:0 XM:i:0 XO:i:0 XG:i:0 MD:Z:50
chr17_123_290_3:0:0_1:0:0_27b3b 163 chr17 123 60 50M = 241 168 ATAACAAACATATGTCCAGCGAATACCTGCATCCCTAGAAGTGAAGCGAC 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:3 SM:i:25 AM:i:25 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:0T10C35C2
chr17_15_201_1:0:0_1:0:0_209a6 83 chr17 152 60 50M = 15 -187 CATCCCTAGAAGTGAAGCCACCGCCCAAAGACACGCCCATATCCAGCTTA 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:23 AM:i:23 X0:i:1 X1:i:1 XM:i:1 XO:i:0 XG:i:0 MD:Z:40G9
chr17_164_380_1:0:0_0:0:0_aa5e4 99 chr17 164 60 50M = 331 217 TGAAGCCACCGCCCAATGACACGCCCATGTCCAGCTTAACCTGCATCCCT 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:37 AM:i:37 X0:i:1 X1:i:0 XM:i:1 XO:i:0 XG:i:0 MD:Z:16A33
chr17_45_242_0:0:0_1:0:0_50f5f 147 chr17 193 60 50M = 45 -198 TCCAGCTTAACCTGCATCCCTAGAAGGGAAGGCACCGCCCAAAGACACGC 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:37 AM:i:37 X0:i:1 X1:i:0 XM:i:1 XO:i:0 XG:i:0 MD:Z:26T23
chr17_204_401_0:0:0_0:0:0_43ea1 99 chr17 204 60 50M = 352 198 CTGCATCCCTAGAAGTGAAGGCACCGCCCAAAGACACGCCCATGTCCAGC 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:0 SM:i:23 AM:i:23 X0:i:1 X1:i:1 XM:i:0 XO:i:0 XG:i:0 MD:Z:50
chr17_224_415_1:0:0_1:0:0_a2d53 163 chr17 224 60 50M = 366 192 GCACCGCCCAAAGACACGCCCATGTCCAGCTTATTCTCCCCAGTTCCTCT 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:37 AM:i:37 X0:i:1 X1:i:0 XM:i:1 XO:i:0 XG:i:0 MD:Z:37G12
chr17_123_290_3:0:0_1:0:0_27b3b 83 chr17 241 60 50M = 123 -168 GCCCATGTCCAGCTTATTCTGCCCAGTTCCTCTCCAGATAGGCTGCATGG 22222222222222222222222222222222222222222222222222 XT:A:U NM:i:1 SM:i:37 AM:i:25 X0:i:1 X1:i:0 XM:i:1 XO:i:0 XG:i:0 MD:Z:38A11

Best Wishes,
Umer