Q:

Stacking multidimensional numpy arrays with one different dimension

Alright - assume I have two numpy arrays, shapes are:
(185, 100, 50, 3)

(64,  100, 50, 3)

The values contained are 185 or 64 frames of video (for each frame, width is 100 pixels, height is 50, 3 channels, these are just images.  The specifics of the images remain constant - the only value that changes is the number of frames per video)
I need to get them both into a single array of some shape like
(2, n, 100, 50, 3)

Where both videos are contained (to run through a neural net as a batch)
I've already tried using np.stack - but I get 
ValueError: all input arrays must have the same shape

A:

This is a quick brainstorm idea that I've got, along with strategy and Python code.  Note: I was going to stick to just comment but to illustrate this idea I'd need to type in some codes.  So here we go! ( grab a coffee / a strong drink is recommended...)
 Current State

we have video 1 vid1 with 4D shape (185, 100, 50, 3)
we have video 2 vid2 with 4D shape (64, 100, 50, 3)
... where the shape represents (frame ID, width, height, RGB channels)
we want to "stack" the two videos together as one numpy array with 5D shape (2, n, 100, 50, 3).  Note: 2 because we are stacking 2 videos.  n is a hyperparameter that we can choose.  We keep the video size the same (100 width x 50 height x 3 RGB channels)

 Opportunities
The first thing I see is that vid1 has roughly 3 times more frames than vid2.  What about we use 60 as the common factor?  i.e. let's set our hyperparameter n to 60. ( Note: some "frame cropping" / "frame throwing away" may be required - this will be covered below.) 
 Strategy
 Phase 1 - Crop both videos (throw away some frames)
Let's crop both vid1 and vid2 to nice round numbers that are of multiple of 60 (our n - the hyperparameter).  Concretely:

crop vid1 so that the shape becomes (180, 100, 50, 3). ( i.e. we throw away the last 5 frames).  We call this new cropped video vid1_cropped. 
crop vid2 so that the shape becomes (60, 100, 50, 3). ( i.e. we throw away the last 4 frames).  We call this new cropped video vid2_cropped. 

 Phase 2 - Make both videos 60 frames

vid2_cropped is already at 60 frames, with shape (60, 100, 50, 3).  So we leave this alone. 
vid1_cropped however is at 180 frames.  So I suggest we reduce this video to 60 frames, by averaging the RGB channel values in 3-frame batches - for all pixel positions (along width and height).  What we will get at the end of this process, is a somewhat "diluted" (averaged) video with the same shape as vid2_cropped - (60, 100, 50, 3).  Let's called this diluted video vid1_cropped_diluted. 

Phase 3 - stack the two same-shape videos together
Now that both vid2_cropped and vid1_cropped_diluted are of the same 4D shape (60, 100, 50, 3).  We may stack them together to obtain our final numpy array of 5D shape (2, 60, 100, 50, 3) - let's call this vids_combined. 
We are done! 
Demo
Turning the strategy into codes.  I did this in Python 3.6 (with Jupyter Notebook / Jupyter Console). 

Some notes:

I yet to validate the code (and revised as needed).  In the meantime If you see any bugs please shout - I will be happy to update. 

I have a gut feeling line 10 below on "diluting" (np.average step) might contain error.  i.e. I mean to perform the 3-frame averaging only against the RGB channel values, for all pixel positions.  I need to double check syntax. ( In the meantime please kindly check line 10!) 

this post illustrates concepts and some code implementation.  Ideally I would have step through this in more depth, via much smaller video sizes so we may obtain better intuition / visualise each step, pixel by pixel. ( I might come back to this when I have time).  For now, I believe the numpy array shape analysis is sufficient to convey the idea across. 

In [1]: import numpy as np

In [2]: vid1 = np.random.random((185, 100, 50, 3))

In [3]: vid1.shape
Out[3]: (185, 100, 50, 3)

In [4]: vid2 = np.random.random((64, 100, 50, 3))

In [5]: vid2.shape
Out[5]: (64, 100, 50, 3)

In [6]: vid1_cropped = vid1[:180]

In [7]: vid1_cropped.shape
Out[7]: (180, 100, 50, 3)

In [8]: vid2_cropped = vid2[:60]

In [9]: vid2_cropped.shape
Out[9]: (60, 100, 50, 3)

In [10]: vid1_cropped_diluted = np.average(vid1_cropped.reshape(3,60,100,50,3),
       : axis=0)

In [11]: vid1_cropped_diluted.shape
Out[11]: (60, 100, 50, 3)

In [12]: vids_combined = np.stack([vid1_cropped_diluted, vid2_cropped])

In [13]: vids_combined.shape
Out[13]: (2, 60, 100, 50, 3)

