Yes, you can omit the walnuts. They do add depth of flavor and texture to the loaf but the final product won't be altered too much without nuts.