You are right: Japanese beetles love to dine on four o'clocks, and according to several university sources these plants are poisonous to them. They are also toxic to people and pets. They can cause vomiting and diarrhea if eaten and the sap can cause dermatitis.