This is not a common occurrence for lovebirds. Your new bird may be frightened of you or of his new cage. The new smells, food, lights and sounds can confuse and distress him. Give him a day or two to adjust to his surroundings. Don't put any toys in the cage for the first few days.
New toys can make the baby frightened and aggressive.
Add one toy at a time later. Too many toys will confuse and frighten him. The beak is a third hand to any parrot. He will use his beak to climb, grab and reach. When he steps onto your hand or arm he will use his beak first to steady himself.