Sometimes even craft felt can be incredibly smooth. Take a hairbrush or clean scrub brush and "brush" it a few times to allow some of those fibers to come out of the weave of the felt. If that doesn't work you can always use small velcro pieces (the hook half not the loop half) or some double-sided tape.