Magnets are just not strong enough so I have added another 6 to solve the issue.