The chest is locked until you progress through Ranni's questline. In order to open the chest beside Rennala in the Grand Library, you must progress through Ranni's questline.