Many things can degrade the signal, making for poor connections, and thus speed. Not always easy to sort out. Your provider should be able to assist you. First, make sure the problem isn't with your setup as they likely won't help solve a Linux problem. Then be persistent, and don't hesitate to go over someone's head if the help is not forthcoming.
Most problems are solvable.
The trick is isolating it. A good telco tech, trained for DSL, can find all kinds of obscure wiring problems. Q. My provider's tech support staff is clueless. Seems to be the nature of the beast. First line tech support is an entry level position, and mostly filled by young people with little technical or networking knowledge.