set bottleneck to 100
train the top part of the net only, starting from a state with two nice kway paths
start learning rate unusually high so our continuation point starts
with the usual 0.01
bottleneck 100 works better than 50, so use it
try to learn a weighted distance also
reduce learning rate to see if we can avoid explosion
lower learning rate again to try to avoid explosion
just learn the 4 params at the top, but decrease the learning rate twice
init top layers to all zeros
truncate training data to 6400 pairs -- hope to see fast over-fitting
test on the truncated (6400 pairs) training data for now
set solver params for testing on 6400 training examples
train and test on truncated data only
clean house
does kway + closs on a small training set even converge on that?
increase weight on contrastive loss
test less
crank up learning rates even further
reduce base lr
simplify further, make bottleneck bigger
reweight losses