Guess I lied when I said I wasn't going to use Angular for this.
Tried a couple things, figured out that Justified Gallery does exactly what I want I think.
CSS Tweaks
Switched Jquery Gallery plugin
Switched Gallery edit page to a more inline design. Also angular.