Rabbits can't see objects up close, so they use their sense of smell instead. It could be that your fingers smell of pellets or fruit, causing the rabbit to nibble in the expectation of something tasty. In this case, wash your hands thoroughly so that they smell neutral and not of food or perfumes.