Using an unrelated file as a decoy, I discussed Mr. Freid's project without my boss knowing that Mr. Freid was my client and not his. I later told Mr. Freid to call my cell phone with any further inquiries, or to leave a message. There's a price for doing double duty, and you must be prepared to pay it. Your free time disappears. You can give yourself more free time, however, by maximizing the overlap between your job and your freelancing work.
Speed is the issue.
If you are doing the same sort of work in both capacities, you will become very efficient. While I spend eight to nine hours a day at the office, the three to five hours I spend freelancing yield a better return. Think of the hourly rate you would like to maintain and that will tell you if you are working fast enough. Working a day job and being an after hours IP requires a willingness to tough it out over a long period of time. Well, let me tell you a story.