Yes, any amount of petrol can cause serious damage to a diesel car.
If you spot your mistake just after starting to fill up, hang up on the pump and do not start the engine. Move your car to a safe place and call for misfuelling help.