Fix LSM trees with custom collators: use the same customize path as for btrees

refs #1361