This isn't ideal for scaling.  At some point this is going to get huge unless the application history logs are cleaned up.  Perhaps we leave that for a follow on improvement though since I think you are limited by the way its storing them now.  The first thing that came to mind was just to list by date and choose the latest x but hadoop filesystem doesn't give you that filter without doing stat on each one afterwards.