That feature does not work. Here is what fails for me:

Create external partitioned table
add partition
cache table
add partition

the second partition is not in memory; and when I query the table, I get partial and incorrect results. i.e. I only get back incorrect results from the first partition and no error to the user. Its more than a exception/messaging issue.