Yes, you can be allergic to semolina for two reasons. If you are allergic to food made out of wheat then you are allergic to semolina as well. Because semolina is made from durum wheat. Also, be aware that semolina is rich in gluten. If you have gluten intolerance then avoid taking semolina in any form. More here on health risks and considerations.