Setting up new IB+Carette mutant analysis code - NOT WRITTEN.
Minor: moved plotting functions from mutant_Carette to mutant_plotting_utilities.
MID-REWRITE: changed options, removed unneeded stuff, reformatting.

Updated all the options - included new ones, removed deprecated ones or
unnecessary ones or ones that should be in a separate function.
Changed the --read_direction (forward/reverse compared to cassette)
option to --relative_read_direction (inward/outward to the cassette) -
this way I don't have to change it between 5' and 3' ends.

Removed everything related to mutant-merging and to parsing metadata
files - those things are going to be pretty different for the new
IB+RISCC pipeline, may write them later.

Removed everything related to removing mutants based on other files -
may add later as a separate program.

Moved test functions to the end; modified main to take the right args.

THIS DOESN'T WORK RIGHT NOW - I only did the "skeleton" of the rewrite.
MID-REWRITE: skeleton of new IB+RISCC classes, not finished!!

Merged code from mutant_analysis_classes.py and mutant_Carette.py into
new IB+RISCC data (Carette is renamed to RISCC) - DOESN'T WORK YET, I
didn't do most of the implementing!

Rewrote get_insertion_pos_from_flanking_region_pos to use
relative_read_direction instead of reads_are_reverse.

Started rewriting Insertional_mutant to use barcodes - not sure if
that's done.

Need to rewrite all of Insertional_mutant_dataset to be based on
barcodes!!

Removed mutant-merging for now - may re-add later, but it'll have to be
very different.

Removed deprecated text-file-reading methods - sticking with .pickle
files only from now on.

Minor: added some constants; moved some sections around; renamed Carette
to RISCC etc.
Refactored get_RISCC_pos_from_read_pos; renamed; comments.

Rewrote get_RISCC_pos_from_read_pos (renamed from
get_RISCC_pos_from_flanking_region_pos) to use
get_insertion_pos_from_flanking_region_pos instead of copying most of
its code.

NOT TESTED YET.
Added confirmed/wrong counts to best-only (space-saving) option

Previously, when adding Carette genome-side data to a dataset using the
best_only option (using the
Insertional_mutant_Carette.improve_best_Carette_read function), you'd
get just the single best read (giving the max confirmed distance), but
you WOULDN'T get the counts of confirming and non-confirming reads,
which are the other important basic bit of info.
I added them, in a sort of messy way.

Normally these read-counts are functions or properties, since they can
be derived from the actual stored reads. In the best-only variant, only
one read is ever stored, so the read-counts have to be normal variables.
I made the Insertional_mutant_Carette.improve_best_Carette_read function
replace the property (self.Carette_N_unaligned_reads) with a variable,
but I couldn't come up with a good way of replacing the non-property
functions (self.Carette_N_confirming_reads and
self.Carette_N_non_confirming_reads), so instead I changed them to
return private variables IF they exist, and made the function create
those variables.

Tested on real data, seems sane.
IN PROGRESS: most of the new IB+RISCC implementation.

Changed "barcode" to "IB" everywhere - that should've been in a separate
commit, sorry.

Changes to Insertional_mutant class:
 - moved read_info
 - docstring, comments, etc
 - changed the genome-side reads from a full mutant object list to a
   seq:(pos, readcount, N_errors, gene, annotation...) dict
 - rewrote add_RISCC_read and improve_best_RISCC_read to always just
   take the seq and position, not the alignment, and fill out the new
   dict based on that
 - modified various properties and descriptive functions to use the new
   dict

Changes to Insertional_mutant_pool_dataset class:
 - changed the main mutant-containing data structure to be
   _mutants_by_IB rather than _mutants_by_position; rewrote the basic
   container-type functions to use that.
 - removed get_matching_position_mutants entirely, for now
 - wrote add_alignment_files_to_data (plus help functions
   _next_until_name_match and _parse_3files_parallel, copied from
   experiments/arrayed_library/internal_barcode_processing/code/clustering_tools.py
   with some modifications) to replace add_alignment_reader_to_data
   and add_RISCC_genome_side_alignments_to_data (which were separate
   functions for cassette-side and genome-side reads - now both are done
   in parallel).  The new function uses some code from the old ones.
   IT'S NOT QUITE FINISHED - see TODO comments.
 - rewrote the gene-finding and annotation-adding functions to work with
   new per-mutant genome-side read dict that isn't a list of mutants
 - comments

Not done!  Need to finish add_alignment_files_to_data, and do LOTS of
testing and I'm sure fix lots of details.
Removed mutant.read_IDs - no longer needed.

Used to be needed to keep the read IDs from the cassette-side reads to
later add the matching genome-side ones, but now both are done in
parallel (which should save memory/time), so no need to save them. They
were pretty memory-wasteful in datasets with a lot of reads!
Switched to simpler/safer way of checking #errors in HTSeq alignments.

Previously used check_mutation_count_try_all_methods, which was slower
(checked CIGAR string, NM field and MD field, compared all) and less
safe (returned -1 rather than raising an error if the data was unclear).
Switched to check_mutation_count_by_optional_NM_field with
negative_if_absent=False, so that it only checks the NM field (we know
bowtie always generates it) and raises an error if not found, so we can
deal with weird cases if they happen.
Removed the treat_unknown_as_match option everywhere, since it's only
relevant to CIGAR strings.
Separated test data for old and new code.

Added test_data_v0 folder, with copies of all the files from test_data
that are used by mutant_analysis_classes.py and
mutant_count_alignments.py, since these two are now old files that are
being replaced by mutant_IB_analysis.py and mutant_count_aln_IB.py
(mutant_Carette.py is also being replaced but it had no test files).
Renamed main new files to include RISCC in name along with IB.
Added detection of unaligned/multi positions, and mutant position check.

Wrote parse_flanking_region_aln_or_pos function and modified
get_insertion_pos_from_flanking_region_pos and ISCC_pos_from_read_pos so
that they can take an HTSeq alignment as input and return special
unaligned/multi-aligned position codes in the right cases (but they can
also still take simple position tuples for when I need that).  Modified
Insertional_mutant_pool_dataset.add_alignment_files_to_data to use that
properly - it's basically finished now.

ALSO modified Insertional_mutant.add_read to take a position argument
and check that the position matches the current one (unless current one
is unspecified) - this is needed now that reads are clustered into
mutants based on IB and not alignment position, so it's possible that
mutants with the same IB could have different alignment position, and we
want to check these cases.  May need to add more complicated handling
later (for things like +/-1bp due to deepseq/PCR indels).
Changed reads_are_reverse to relative_read_direction everywhere.

New version should run on relative_read_direction rather than
reads_are_reverse - this is different, but the actual difference was
already implemented in get_insertion_pos_from_flanking_region_pos and
get_RISCC_pos_from_read_pos, now I just needed to change the names,
expected values, etc.

Also added some section headers.

DIDN'T change the unit-tests yet.
Rewrote further to use new mutant_count_aln_IB+RISCC.py version.

Rewrote command-line options to match what the
add_alignment_reader_to_data function in mutant_count_aln_IB+RISCC.py
needs.
Removed the --ignore_cassette and --separate_cassette options for
simplicity for now.

Rewrote main to match new mutant_count_aln_IB+RISCC.py code - some stuff
still not implemented properly (adjacent mutant counting,
gene/annotation info), but the basics should be there.

Probably still not functional but getting close.
Renamed add_RISCC_alignment_files_to_data; other minor.

Renamed add_alignment_files_to_data to
add_RISCC_alignment_files_to_data, since there should also be an
add_IB_only_alignment_files_to_data function in the future.  Added
comments/TODOs, fixed some docstrings, etc.
Partial option/code fixes for gene/annotation data - still not done.
Starting unit-tests: some fixes and test additions.

Added unit-test for parse_flanking_region_aln_or_pos, fixed it to pass.

Fixed some syntax stuff.

Turned all non-rewritten unit-tests off to avoid having a huge amount of
failures to go through - will turn them on one by one as I work on them.
Position-stuff test, bugfixes, refactoring. IN PROGRESS.

Wrote unit-tests for get_insertion_pos_from_flanking_region_pos;
refactored some of the previous unit-tests.

Added check_valid_position_tuple and used it in the other position-based
functions. Added extra sanity-check to parse_flanking_region_aln_or_pos
(checks if flanking_region_aln_or_pos HAS an .iv attribute before using
it), refactored. Fixed typo in get_insertion_pos_from_flanking_region_pos.

Fixed tests pass; other tests still disabled and probably failing.
Added get-RISCC-pos test; refactoring/fixes; moving tests around.

Added check_valid_end_info, refactored other get-position functions with
it.  Added unit-testa for get_RISCC_pos_from_read_pos - pass.

Went over the other old unit-tests and moved them around or removed as
appropriate to make them match the order of stuff in the file.
Mutant __init__ test; added IB to multi-dataset mutant init.