First pass cleanup for RNN building