## 1 Introduction  
Advancement of recent information technologies evolves the agriculture and food industries. Enterprises are eager to create smart information systems to increase their business competition. Coffee is the largest amount of raw-food material in the world-wide agricultural trade [1]. For example, coffee beans worth over 25 billion USD per year are imported to the Taiwan market. Figure 1 illustrates the production process from green beans to coffee bean packages. The process illustrates from coffee fruit, green beans, defective bean removal, roast beans, to the final bean package. The classes of defective beans are categorized by the SCAA (Specialty Coffee Association of America). Among these stages, defective bean removal (or in short, defect removal) is one of most labor-consuming stages, and many companies investigate the automation of this stage for minimizing human efforts. In addition, the defective bean removal is a critical stage that affects the bean's value, as many experts explicitly point out that defective beans are a key factor for providing high-quality coffee. Thus, removing defective beans becomes a necessary step before brewing for significantly increasing their competition and profits [2]. The SCAA has classified the defective beans into 13 classes [3], as shown in upper portion of Figure 1. Most popular defective bean removal processes are achieved by manual or mechanical manners in the past decades [1; 2; 4; 5]. Nevertheless, these solutions focus on removing few sorts of defects in the SCAA's classification. If all defects need to be removed completely, the cost of purchasing these products is too expensive. With the advance of robotic techniques, all defective beans can be completely picked off with only one arm equipment, which greatly reduces working space and financial cost. By surveying related literature [6; 7; 8; 9; 10; 11], most recent works solve this issue with the vision technologies with artificial intelligence. The difficulties of identifying defective beans in real-world working spaces includes bean overlapping, different surface sizes of beans, different degrees of defective appearances, etc. [PERSON] et al. [12] employed an artificial neural network (ANN) to automatically categorize the coffee beans of four selected locations according to their provenance. They considered certain properties (color, morphology, texture, and a combination of morphology and color features), which cover only some of defective beans. For providing a uniform solution to industries, computational intelligent technologies, such as pattern mining [13; 14; 15] or machine/deep learning [3; 16; 17; 18; 19], bring considerable techniques to develop actionable analytics and prediction to  
Figure 1: An illustration of production process from green beans to coffee bean packages. completely detect all sorts of detective bean patterns at the same time. Deep-learning-based models are inherently more suitable for accomplishing complex tasks with enormous data inputs than generic data-analytic-based models. However, none of them consider the data labeling cost in their solutions, so there is a great barrier to lead these methods to industries. This motivated this work to investigate the issue of identifying all defective beans with less labor effort. Traditional data labeling over beans may be performed in three ways [3; 16; 18]. The first one is \"one-by-one labeling\", in which the user takes pictures for an individual bean and labels the bean for these pictures. This make a coffee always in a specific position or in a blank background, which lead the imprecision inspection in the practice due to the high variance of the trained model. The second one is \"all-in-one labeling\", in which the user takes a picture of the sufficient amount of beans in a container (e.g., a plate) and labels these beans one by one on a screen. This shall perform better than the first way since it reduces the high variance issue.