Fix nested arrays when using numeric keys

> This expands on #2680 specifically @asiral's comment to join nested arrays that are numerically keyed.

Originally opened as #4590 re-PRed against 4.1 as requested by @GrahamCampbell