arq5x/bedtools — Issue #10: bedtools git archive is very large due to old binaries and data files
Author: bpow
Created: 2011-12-20 20:23:42+00:00
State: open

There are a lot of rather large files that had at one time been committed to the git archive and subsequently deleted. Because git keeps track of these old objects, the archive is ~467 mb on despite the current files being less than about 6mb.

Some of the files and their uncompressed file sizes) that could be removed from the history include:

1.3M data/knownPhenotypes.bed (7110362: 1 year, 12 months ago)
1.3M src/intersectBed/chr1_snp129.bed (f50878e: 2 years, 7 months ago)
1.4M src/intersectBed/new.out (f50878e: 2 years, 7 months ago)
1.6M BEDTools-User-Manual.pdf (b854c68: 1 year, 11 months ago)
1.6M src/pairToBed/test (620895f: 1 year, 12 months ago)
1.7M BEDToolsManual.v2.4.0.pdf (e7388c6: 1 year, 11 months ago)
1.7M data/DNA.lte200MilliDiv.mm9.bed (7110362: 1 year, 12 months ago)
1.9M src/intersectBed/origTest (4034c4c: 2 years, 8 months ago)
2.4M src/pairToBed/neither.ids (323b6f7: 1 year, 12 months ago)
2.6M data/SINES.chr14.mm9.bed (7110362: 1 year, 12 months ago)
2.6M src/pairToBed/xor (323b6f7: 1 year, 12 months ago)
4.4M scripts/yeast.gtf (8e0db9f: 2 years, 1 month ago)
4.6M data/dnaMER1.mm9.bed (7110362: 1 year, 12 months ago)
5.9M data/microSatellites.mm9.track.bed (7110362: 1 year, 12 months ago)
6.9M data/dnaTransposons.bed (7110362: 1 year, 12 months ago)
7.8M data/DNAS.mm9.bed (7110362: 1 year, 12 months ago)
10.4M data/snps.hg18.chr21.v130.bed (9a8611d: 1 year, 11 months ago)
11.7M data/segDupsLiftedFromMM8.bed (7110362: 1 year, 12 months ago)
16.7M data/LTRS.lte200MilliDiv.mm9.bed (7110362: 1 year, 12 months ago)
17.3M src/intersectBed/hmec.sam.tier1.bam (323b6f7: 1 year, 12 months ago)
18.9M data/LINES.lte200MilliDiv.mm9.bed (7110362: 1 year, 12 months ago)
19.0M src/pairToBed/hmec.txt.pairs.bam (323b6f7: 1 year, 12 months ago)
19.7M src/pairToBed/simrep.hg18.txt (323b6f7: 1 year, 12 months ago)
22.6M data/SINES.lte200MilliDiv.mm9.bed (7110362: 1 year, 12 months ago)
23.9M src/b6j.test.bed (0a67b92: 2 years, 1 month ago)
30.1M data/allSimpleRepeats.mm9.bed (7110362: 1 year, 12 months ago)
35.7M src/intersectBed/simplerepeats.mm8.bed (932788f: 1 year, 12 months ago)
36.6M data/simpleReps.mm9.bed (7110362: 1 year, 12 months ago)
39.3M data/LTRS.mm9.bed (7110362: 1 year, 12 months ago)
41.9M data/LINES.mm9.bed (7110362: 1 year, 12 months ago)
51.5M src/intersectBed/old (932788f: 1 year, 12 months ago)
53.6M src/pairToBed/neither (323b6f7: 1 year, 12 months ago)
56.6M src/coverageBed/old (82b7108: 1 year, 12 months ago)
56.6M src/coverageBed/old.sorted (82b7108: 1 year, 12 months ago)
59.9M data/allTransposons.lte200MilliDiv.bed (7110362: 1 year, 12 months ago)
65.9M data/simpleRepeats.mm9.fromRepeatMasker.bed (7110362: 1 year, 12 months ago)
66.9M data/SINES.mm9.bed (7110362: 1 year, 12 months ago)
69.9M src/intersectBed/hmec.merged.bam (db3b2cd: 1 year, 12 months ago)
123.1M data/allTransposons.bed (7110362: 1 year, 12 months ago)
138.3M data/rnaTransposons.bed (7110362: 1 year, 12 months ago)
170.0M scripts/mouse.gtf (8e0db9f: 2 years, 1 month ago)
242.4M data/repeatMasker.mm9.bed (7110362: 1 year, 12 months ago)
278.3M data/repMaskerAll.txt (7110362: 1 year, 12 months ago)

These files could be removed from the history following instructions at http://http://help.github.com/remove-sensitive-data/

The downside of doing this would be a need to rebase (or perhaps it would be easiest to move the old repository to a different name and make a new repository without the above files). The upside would be a much smaller git archive,