You may be terrified inside, but they are under the impression that you know something they don't. You have an edge that doesn't quite make sense to them. You can call that edge whatever you want but the reality is that its arrogance. Watch successful directors at work, in interviews, anywhere except in their personal lives - they are the most arrogant people on the planet. Of course they're not really, but that's the whole point.
Art direction is another skill that tends to get ignored in the world of low budget filmmaking.
Few film schools even take it seriously enough to run a proper program for art directors. Hence when it comes to assigning the production roles it is usually the lost souls or the dead wood that ends up being shoved in the art department. Your art director should be as important as your director of photography. They will both define the look of your film. It's no use having a talented D.O.P. if you are then going to ask them to light a set or a location that hasn't been thought out properly first.