Sounds like one of two things happened: first, be sure to let it rest overnight in the fridge. If you rush this step, the residual heat will prevent the dessert from setting up. Second, be sure to use enough agar agar. If you use too little, the panna cotta will end up being more like a pudding. Don't worry though, it'll still taste good!