title: Studying and exploiting the relationship between model accuracy and explanation quality

authors: Jia, Yunzhe

subjects: computer science

abstract: Many explanation methods have been proposed to reveal insights about the internal procedures of black-box models like deep neural networks. Although these methods are able to generate explanations for individual predictions, little research has been conducted to investigate the relationship of model accuracy and explanation quality, or how to use explanations to improve model performance. In this paper, we evaluate explanations using a metric based on area under the ROC curve (AUC), treating expert-provided image annotations as ground-truth explanations, and quantify the correlation between model accuracy and explanation quality when performing image classifications with deep neural networks. The experiments are conducted using two image datasets: the CUB-200-2011 dataset and a Kahikatea dataset that we publish with this paper. For each dataset, we compare and evaluate seven different neural networks with four different explainers in terms of both accuracy and explanation quality. We also investigate how explanation quality evolves as loss metrics change through the training iterations of each model. The experiments suggest a strong correlation between model accuracy and explanation quality. Based on this observation, we demonstrate how explanations can be exploited to benefit the model selection process—even if simply maximising accuracy on test data is the primary goal.

text: Studying and exploiting the relationship between model accuracy and explanation quality Many explanation methods have been proposed to reveal insights about the internal procedures of black-box models like deep neural networks. Although these methods are able to generate explanations for individual predictions, little research has been conducted to investigate the relationship of model accuracy and explanation quality, or how to use explanations to improve model performance. In this paper, we evaluate explanations using a metric based on area under the ROC curve (AUC), treating expert-provided image annotations as ground-truth explanations, and quantify the correlation between model accuracy and explanation quality when performing image classifications with deep neural networks. The experiments are conducted using two image datasets: the CUB-200-2011 dataset and a Kahikatea dataset that we publish with this paper. For each dataset, we compare and evaluate seven different neural networks with four different explainers in terms of both accuracy and explanation quality. We also investigate how explanation quality evolves as loss metrics change through the training iterations of each model. The experiments suggest a strong correlation between model accuracy and explanation quality. Based on this observation, we demonstrate how explanations can be exploited to benefit the model selection process—even if simply maximising accuracy on test data is the primary goal.

year: 2021

title: The Environmental and Economic Impacts for Producing the Port Jackson from Novatein®

authors: van der Merwe, De Wet

subjects: LCA

abstract: Preserving meat quality is paramount during meat processing. Contamination is reduced by preventing contact between faecal matter and the carcass. Rectal plugs are used for this purpose during slaughtering, and the animal’s intestines can be removed and entered directly into the rendering process. Polypropylene plugs contaminate rendered products, while Novatein® plugs (the Port Jackson) will break down during the rendering process. 

A life cycle assessment (LCA) was used to determine and compare the environmental impacts of plugs made from either polypropylene or Novatein®. Two scenarios were considered, in which the production of the plugs could be contracted out, or manufactured in-house. On an environmental basis, the difference was negligible. Resin production accounted for 73% of the total non-renewable primary energy (NRPE) use and global warming potential (GWP), with the balance shared between plug production, use, and disposal. 

Injection moulding and packaging each contributed 31% to GWP, as well as 25.8% and 40% of the NRPE use respectively during plug production and disposal, dominating impacts during plug production. In contrast, operations with the lowest impacts were conditioning and transport, collectively contributing to less than 4% of the entire LCA impacts. 

The life cycle impacts were particularly sensitive to packaging, the allocation method used for farming impacts on bloodmeal production, and the ratio of the electricity grid mix. By eliminating cardboard boxes reduced GWP and NRPE use by 6.1% and 6.9%. Allocating impacts from farming to bloodmeal production could increase the GWP by up to 193%, and NRPE by 14%. Allocation is not under the control of the manufacturer, and is a limitation of the assumptions made in this study. By using coal-based electricity, the GWP can increase by 66%, and NRPE by 20%. Impacts from electricity use could change noticeably if the plugs were manufactured overseas. 

The net present value (NPV) of capturing 10% of the market, and selling the plugs at $0.15 NZD, at a 35% discount rate was $143,629 when contracting out plug production. Under all costing assumptions, contracting out provided a higher NPV than manufacturing in-house. This is primarily due to the large capital costs when producing in-house. 

The critical factor that will cause manufacturing in-house to be more financially viable is the capital cost. The difference between the NPV of the two scenarios is $240,442, and the estimated total cost of buying, shipping, and installing the injection moulder is $355,153. If this can be reduced to $114,711, the two scenarios become equal. Because manufacturing in-house has a higher ongoing cash inflow per year, this scenario will also benefit more from scaling up volume above the 10% market share threshold. 

When comparing the Port Jackson with the polypropylene plug, the Novatein® plug has a higher GWP (0.0166 kg CO₂eq per plug) than the PP plug (0.0126 kg CO₂eq per plug), however, it requires less NRPE (0.302 MJ per plug and 0.430 MJ per plug respectively), and if contracted out, can be sold competitively at a matching market value of $0.15 or $0.16 per plug. The price may be lowered even further to $0.12, and still have a positive NPV. These factors are slightly in favour of producing Novatein plugs, but the Port Jackson’s advantage over PP is the fact that it not only breaks down during rendering, but it is also non-toxic, so simply becomes part of part of meat and bone meal.

text: The Environmental and Economic Impacts for Producing the Port Jackson from Novatein® Preserving meat quality is paramount during meat processing. Contamination is reduced by preventing contact between faecal matter and the carcass. Rectal plugs are used for this purpose during slaughtering, and the animal’s intestines can be removed and entered directly into the rendering process. Polypropylene plugs contaminate rendered products, while Novatein® plugs (the Port Jackson) will break down during the rendering process. 

A life cycle assessment (LCA) was used to determine and compare the environmental impacts of plugs made from either polypropylene or Novatein®. Two scenarios were considered, in which the production of the plugs could be contracted out, or manufactured in-house. On an environmental basis, the difference was negligible. Resin production accounted for 73% of the total non-renewable primary energy (NRPE) use and global warming potential (GWP), with the balance shared between plug production, use, and disposal. 

Injection moulding and packaging each contributed 31% to GWP, as well as 25.8% and 40% of the NRPE use respectively during plug production and disposal, dominating impacts during plug production. In contrast, operations with the lowest impacts were conditioning and transport, collectively contributing to less than 4% of the entire LCA impacts. 

The life cycle impacts were particularly sensitive to packaging, the allocation method used for farming impacts on bloodmeal production, and the ratio of the electricity grid mix. By eliminating cardboard boxes reduced GWP and NRPE use by 6.1% and 6.9%. Allocating impacts from farming to bloodmeal production could increase the GWP by up to 193%, and NRPE by 14%. Allocation is not under the control of the manufacturer, and is a limitation of the assumptions made in this study. By using coal-based electricity, the GWP can increase by 66%, and NRPE by 20%. Impacts from electricity use could change noticeably if the plugs were manufactured overseas. 

The net present value (NPV) of capturing 10% of the market, and selling the plugs at $0.15 NZD, at a 35% discount rate was $143,629 when contracting out plug production. Under all costing assumptions, contracting out provided a higher NPV than manufacturing in-house. This is primarily due to the large capital costs when producing in-house. 

The critical factor that will cause manufacturing in-house to be more financially viable is the capital cost. The difference between the NPV of the two scenarios is $240,442, and the estimated total cost of buying, shipping, and installing the injection moulder is $355,153. If this can be reduced to $114,711, the two scenarios become equal. Because manufacturing in-house has a higher ongoing cash inflow per year, this scenario will also benefit more from scaling up volume above the 10% market share threshold. 

When comparing the Port Jackson with the polypropylene plug, the Novatein® plug has a higher GWP (0.0166 kg CO₂eq per plug) than the PP plug (0.0126 kg CO₂eq per plug), however, it requires less NRPE (0.302 MJ per plug and 0.430 MJ per plug respectively), and if contracted out, can be sold competitively at a matching market value of $0.15 or $0.16 per plug. The price may be lowered even further to $0.12, and still have a positive NPV. These factors are slightly in favour of producing Novatein plugs, but the Port Jackson’s advantage over PP is the fact that it not only breaks down during rendering, but it is also non-toxic, so simply becomes part of part of meat and bone meal.

year: 2014

title: Boat electrofishing survey of fish abundance in the Ohau Channel, Rotorua, in 2014

authors: Hicks, Brendan J.

abstract: The aim of the survey was to provide on-going monitoring of the fish communities and abundance by boat electrofishing in the Ohau Channel, especially fish species that are taonga to Maori (eels, goldfish, and koura). In the current study, we present the findings from the eighth year of sampling (2014) and a summary of previous surveys.
We used the University of Waikato’s 4.5 m-long, aluminium-hulled electrofishing boat to catch a total of 642 fish (11.3 kg) 11 sites on 9 December 2014, which comprised 2,914 lineal m and 11,656 m² in area. Koura and 4 fish species were present, with common bully most abundant (up to 27 fish 100 m⁻² at the site 5, edge habitat). Goldfish (up to 5.92 g m⁻²) was the next most abundant species, and was most abundant in sites 8-11 the lower channel, especially at site 11, the excavated side channel. Rainbow trout were next the most abundant species (up to 0.25 fish 100 m⁻² ). Mean bully biomass (5.25 g m⁻²) was much higher than for smelt (0.07 g m⁻²). Koura had a patchy distribution; 5 individuals were caught.
Comparing catches over the 8 years of sampling, the abundance of common bullies in 2014 (4.81 fish 100 m⁻²) was consistent with catch rates in most post-wall years (after 2007; range 0.81-5.72 fish 100 m⁻²), but lower than in 2013. The cause of fluctuating bully abundance is not known, and was not accounted for by changes in water clarity expressed as black disc distance (BDD), water temperature, or water conductivity. Poor water clarity can reduce the efficiency of electrofishing, suggesting that bully density estimates might decrease with decreasing BDD. However, common bully densities in 2012 and 2011 were inversely related BDD. In 2014, catches of smelt, a similar sized fish to bullies, were extremely low.
Goldfish biomass increased initially 2009 and 2010 because of targeted fishing in the excavated side channel (site 11), which has dense macrophyte beds that provide good habitat for goldfish. The continued rise in goldfish density from 2012 on suggests a real increase in goldfish numbers has occurred. Eels were scarce in the Ohau Channel and catches were very variable. Longfin eels were caught in most years, and in 2012 and 2013 shortfin eels were also caught, but in 2014 no eels were caught.
Analysis of fish densities before and after wall closure is hampered by the single data point before closure. However, we now have 7 years of post-wall data, and comparison of means and standard deviations suggest that the number of bullies has decreased.
An intriguing trend of decreasing rainbow trout densities with increasing BDD, which is a measure of both water clarity and phytoplankton abundance, has occurred following wall construction (Figure 3). The trend is opposite to the usual decline of catch rate by boat electrofishing with reduced water clarity. The explanation for this is not immediately clear, and could be increased avoidance of the electrofishing boat by trout in clearer water. It is also possible that reduced BDD indicates high algal productivity in Lake Rotorua, which in turn could increase trout abundance in the Ohau Channel. This remains to be tested

text: Boat electrofishing survey of fish abundance in the Ohau Channel, Rotorua, in 2014 The aim of the survey was to provide on-going monitoring of the fish communities and abundance by boat electrofishing in the Ohau Channel, especially fish species that are taonga to Maori (eels, goldfish, and koura). In the current study, we present the findings from the eighth year of sampling (2014) and a summary of previous surveys.
We used the University of Waikato’s 4.5 m-long, aluminium-hulled electrofishing boat to catch a total of 642 fish (11.3 kg) 11 sites on 9 December 2014, which comprised 2,914 lineal m and 11,656 m² in area. Koura and 4 fish species were present, with common bully most abundant (up to 27 fish 100 m⁻² at the site 5, edge habitat). Goldfish (up to 5.92 g m⁻²) was the next most abundant species, and was most abundant in sites 8-11 the lower channel, especially at site 11, the excavated side channel. Rainbow trout were next the most abundant species (up to 0.25 fish 100 m⁻² ). Mean bully biomass (5.25 g m⁻²) was much higher than for smelt (0.07 g m⁻²). Koura had a patchy distribution; 5 individuals were caught.
Comparing catches over the 8 years of sampling, the abundance of common bullies in 2014 (4.81 fish 100 m⁻²) was consistent with catch rates in most post-wall years (after 2007; range 0.81-5.72 fish 100 m⁻²), but lower than in 2013. The cause of fluctuating bully abundance is not known, and was not accounted for by changes in water clarity expressed as black disc distance (BDD), water temperature, or water conductivity. Poor water clarity can reduce the efficiency of electrofishing, suggesting that bully density estimates might decrease with decreasing BDD. However, common bully densities in 2012 and 2011 were inversely related BDD. In 2014, catches of smelt, a similar sized fish to bullies, were extremely low.
Goldfish biomass increased initially 2009 and 2010 because of targeted fishing in the excavated side channel (site 11), which has dense macrophyte beds that provide good habitat for goldfish. The continued rise in goldfish density from 2012 on suggests a real increase in goldfish numbers has occurred. Eels were scarce in the Ohau Channel and catches were very variable. Longfin eels were caught in most years, and in 2012 and 2013 shortfin eels were also caught, but in 2014 no eels were caught.
Analysis of fish densities before and after wall closure is hampered by the single data point before closure. However, we now have 7 years of post-wall data, and comparison of means and standard deviations suggest that the number of bullies has decreased.
An intriguing trend of decreasing rainbow trout densities with increasing BDD, which is a measure of both water clarity and phytoplankton abundance, has occurred following wall construction (Figure 3). The trend is opposite to the usual decline of catch rate by boat electrofishing with reduced water clarity. The explanation for this is not immediately clear, and could be increased avoidance of the electrofishing boat by trout in clearer water. It is also possible that reduced BDD indicates high algal productivity in Lake Rotorua, which in turn could increase trout abundance in the Ohau Channel. This remains to be tested

year: 2015